[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786527905902985-13708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 09:45:06 dist-test-slave-jq4t krb5kdc[13719](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 09:45:06 dist-test-slave-jq4t krb5kdc[13719](info): set up 2 sockets
Aug 12 09:45:06 dist-test-slave-jq4t krb5kdc[13719](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260812 09:45:07.250743 13708 mini_kdc.cc:121] Time spent starting KDC: real 1.270s user 0.003s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786527905902985-13708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786527905902985-13708-0/krb5kdc/test-user.keytab.
Aug 12 09:45:07 dist-test-slave-jq4t krb5kdc[13719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527907, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T09:45:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T09:45:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.99.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.99.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.99.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab.
Entry for principal kudu/127.13.99.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab.
I20260812 09:45:07.379976 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786527905902985-13708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786527905902985-13708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786527905902985-13708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.62:41365
--webserver_interface=127.13.99.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab
--principal=kudu/127.13.99.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:42299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.99.62:41365
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:45:07.563511 13737 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:45:07.563773 13737 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:45:07.563817 13737 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:45:07.568325 13737 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 09:45:07.568395 13737 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:45:07.568437 13737 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:45:07.568497 13737 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 09:45:07.568540 13737 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 09:45:07.574352 13737 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:42299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786527905902985-13708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786527905902985-13708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.99.62:41365
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab
--principal=kudu/127.13.99.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.62:41365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.99.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13737
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786527905902985-13708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:45:07.575824 13737 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:45:07.577349 13737 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:45:07.591403 13750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:07.592036 13747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:07.593122 13748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:45:07.593637 13737 server_base.cc:1034] running on GCE node
I20260812 09:45:07.594537 13737 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:45:07.596369 13737 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:45:07.597584 13737 hybrid_clock.cc:648] HybridClock initialized: now 1786527907597554 us; error 49 us; skew 500 ppm
Aug 12 09:45:07 dist-test-slave-jq4t krb5kdc[13719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527907, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:45:07.601604 13737 init.cc:377] Logged in from keytab as kudu/127.13.99.62@KRBTEST.COM (short username kudu)
I20260812 09:45:07.602955 13737 webserver.cc:460] Webserver started at http://127.13.99.62:40699/ using document root <none> and password file <none>
I20260812 09:45:07.604029 13737 fs_manager.cc:359] Metadata directory not provided
I20260812 09:45:07.604140 13737 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:45:07.604665 13737 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:45:07.607656 13737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786527905902985-13708-0/minicluster-data/master-0/data/instance:
uuid: "42db805c686140dbacbf1bfb80404015"
format_stamp: "Formatted at 2026-08-12 09:45:07 on dist-test-slave-jq4t"
server_key: "cc0b63d479cfe72c7566c72cef8dcb7f"
server_key_iv: "1009c2a0fb7beccd205d1df3272031e5"
server_key_version: "encryptionkey@0"
I20260812 09:45:07.608326 13737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786527905902985-13708-0/minicluster-data/master-0/wal/instance:
uuid: "42db805c686140dbacbf1bfb80404015"
format_stamp: "Formatted at 2026-08-12 09:45:07 on dist-test-slave-jq4t"
server_key: "cc0b63d479cfe72c7566c72cef8dcb7f"
server_key_iv: "1009c2a0fb7beccd205d1df3272031e5"
server_key_version: "encryptionkey@0"
I20260812 09:45:07.615275 13737 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260812 09:45:07.619381 13760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:07.621380 13737 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 09:45:07.621625 13737 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786527905902985-13708-0/minicluster-data/master-0/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786527905902985-13708-0/minicluster-data/master-0/wal
uuid: "42db805c686140dbacbf1bfb80404015"
format_stamp: "Formatted at 2026-08-12 09:45:07 on dist-test-slave-jq4t"
server_key: "cc0b63d479cfe72c7566c72cef8dcb7f"
server_key_iv: "1009c2a0fb7beccd205d1df3272031e5"
server_key_version: "encryptionkey@0"
I20260812 09:45:07.621762 13737 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786527905902985-13708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786527905902985-13708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786527905902985-13708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:45:07.644773 13737 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:45:07.649010 13737 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:45:07.649313 13737 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:45:07.666877 13737 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.62:41365
I20260812 09:45:07.666939 13851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.62:41365 every 8 connection(s)
I20260812 09:45:07.668269 13737 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
I20260812 09:45:07.671638 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 13737
I20260812 09:45:07.671898 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786527905902985-13708-0/minicluster-data/master-0/wal/instance
I20260812 09:45:07.674417 13708 external_mini_cluster.cc:1428] Setting key e62149fe53e5cd065f4ced06c5a7e155
I20260812 09:45:07.674525 13855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:45:07.687649 13855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42db805c686140dbacbf1bfb80404015: Bootstrap starting.
I20260812 09:45:07.693635 13855 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 42db805c686140dbacbf1bfb80404015: Neither blocks nor log segments found. Creating new log.
I20260812 09:45:07.695609 13855 log.cc:824] T 00000000000000000000000000000000 P 42db805c686140dbacbf1bfb80404015: Log is configured to *not* fsync() on all Append() calls
I20260812 09:45:07.699043 13855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42db805c686140dbacbf1bfb80404015: No bootstrap required, opened a new log
Aug 12 09:45:07 dist-test-slave-jq4t krb5kdc[13719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527907, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
I20260812 09:45:07.707598 13855 raft_consensus.cc:348] T 00000000000000000000000000000000 P 42db805c686140dbacbf1bfb80404015 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42db805c686140dbacbf1bfb80404015" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 41365 } }
I20260812 09:45:07.708065 13855 raft_consensus.cc:374] T 00000000000000000000000000000000 P 42db805c686140dbacbf1bfb80404015 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:45:07.708178 13855 raft_consensus.cc:729] T 00000000000000000000000000000000 P 42db805c686140dbacbf1bfb80404015 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42db805c686140dbacbf1bfb80404015, State: Initialized, Role: FOLLOWER
I20260812 09:45:07.709198 13855 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42db805c686140dbacbf1bfb80404015 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42db805c686140dbacbf1bfb80404015" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 41365 } }
I20260812 09:45:07.709453 13855 raft_consensus.cc:388] T 00000000000000000000000000000000 P 42db805c686140dbacbf1bfb80404015 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 09:45:07.709601 13855 raft_consensus.cc:482] T 00000000000000000000000000000000 P 42db805c686140dbacbf1bfb80404015 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 09:45:07.709769 13855 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 42db805c686140dbacbf1bfb80404015 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:45:07.711365 13855 raft_consensus.cc:504] T 00000000000000000000000000000000 P 42db805c686140dbacbf1bfb80404015 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42db805c686140dbacbf1bfb80404015" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 41365 } }
I20260812 09:45:07.712034 13855 leader_election.cc:302] T 00000000000000000000000000000000 P 42db805c686140dbacbf1bfb80404015 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 42db805c686140dbacbf1bfb80404015; no voters:
I20260812 09:45:07.712459 13855 leader_election.cc:288] T 00000000000000000000000000000000 P 42db805c686140dbacbf1bfb80404015 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 09:45:07.713061 13862 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 42db805c686140dbacbf1bfb80404015 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 09:45:07.713434 13862 raft_consensus.cc:686] T 00000000000000000000000000000000 P 42db805c686140dbacbf1bfb80404015 [term 1 LEADER]: Becoming Leader. State: Replica: 42db805c686140dbacbf1bfb80404015, State: Running, Role: LEADER
I20260812 09:45:07.713635 13857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:07.680978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53230 (local address 127.13.99.62:41365)
0812 09:45:07.681741 (+ 763us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:07.681761 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:07.683202 (+ 1441us) server_negotiation.cc:407] Connection header received
0812 09:45:07.689953 (+ 6751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:07.689968 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:07.690393 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:07.690655 (+ 262us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:07.695412 (+ 4757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:07.696850 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:07.698912 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:07.699438 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:07.699561 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:07.706013 (+ 6452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:07.706072 (+ 59us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:07.706091 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:07.706149 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:07.709108 (+ 2959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:07.709798 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:07.709805 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:07.709810 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:07.709973 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:07.710334 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:07.710337 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:07.710339 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:07.711098 (+ 759us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:07.711293 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:07.711950 (+ 657us) server_negotiation.cc:299] Negotiation successful
0812 09:45:07.712337 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":239,"threads_started":1}
I20260812 09:45:07.713913 13862 consensus_queue.cc:237] T 00000000000000000000000000000000 P 42db805c686140dbacbf1bfb80404015 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42db805c686140dbacbf1bfb80404015" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 41365 } }
I20260812 09:45:07.714349 13855 sys_catalog.cc:564] T 00000000000000000000000000000000 P 42db805c686140dbacbf1bfb80404015 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 09:45:07.715890 13862 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42db805c686140dbacbf1bfb80404015 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 42db805c686140dbacbf1bfb80404015. Latest consensus state: current_term: 1 leader_uuid: "42db805c686140dbacbf1bfb80404015" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42db805c686140dbacbf1bfb80404015" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 41365 } } }
I20260812 09:45:07.716034 13862 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42db805c686140dbacbf1bfb80404015 [sys.catalog]: This master's current role is: LEADER
I20260812 09:45:07.717473 13864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42db805c686140dbacbf1bfb80404015 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "42db805c686140dbacbf1bfb80404015" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42db805c686140dbacbf1bfb80404015" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 41365 } } }
I20260812 09:45:07.717662 13864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42db805c686140dbacbf1bfb80404015 [sys.catalog]: This master's current role is: LEADER
W20260812 09:45:07.723002 13886 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 42db805c686140dbacbf1bfb80404015: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 09:45:07.723143 13886 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 09:45:07.723227 13876 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 09:45:07.724381 13876 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 09:45:07.733181 13876 catalog_manager.cc:1347] Generated new cluster ID: 766178f2a93a45ff9135be3a8a0146ce
I20260812 09:45:07.733248 13876 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 09:45:07.747082 13876 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 09:45:07.748083 13876 catalog_manager.cc:1504] Loading token signing keys...
I20260812 09:45:07.755079 13876 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 42db805c686140dbacbf1bfb80404015: Generated new TSK 0
I20260812 09:45:07.755959 13876 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.99.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.99.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.99.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.1.keytab.
Entry for principal kudu/127.13.99.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.1.keytab.
I20260812 09:45:07.803231 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786527905902985-13708-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786527905902985-13708-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786527905902985-13708-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786527905902985-13708-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.99.1:0
--local_ip_for_outbound_sockets=127.13.99.1
--webserver_interface=127.13.99.1
--webserver_port=0
--tserver_master_addrs=127.13.99.62:41365
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.1.keytab
--principal=kudu/127.13.99.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:42299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:45:07.957448 13902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:45:07.957692 13902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:45:07.957736 13902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:45:07.961654 13902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:45:07.961704 13902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:45:07.961759 13902 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.99.1
I20260812 09:45:07.966753 13902 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:42299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786527905902985-13708-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786527905902985-13708-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.1.keytab
--principal=kudu/127.13.99.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.99.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786527905902985-13708-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.99.1
--webserver_port=0
--tserver_master_addrs=127.13.99.62:41365
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13902
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.99.1
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786527905902985-13708-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:45:07.967720 13902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:45:07.968696 13902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:45:07.978813 13924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:07.978871 13922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:07.978884 13915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:45:07.979177 13902 server_base.cc:1034] running on GCE node
I20260812 09:45:07.979648 13902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:45:07.980276 13902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:45:07.981436 13902 hybrid_clock.cc:648] HybridClock initialized: now 1786527907981406 us; error 54 us; skew 500 ppm
Aug 12 09:45:07 dist-test-slave-jq4t krb5kdc[13719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527907, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:45:07.984742 13902 init.cc:377] Logged in from keytab as kudu/127.13.99.1@KRBTEST.COM (short username kudu)
I20260812 09:45:07.985596 13902 webserver.cc:460] Webserver started at http://127.13.99.1:35575/ using document root <none> and password file <none>
I20260812 09:45:07.986161 13902 fs_manager.cc:359] Metadata directory not provided
I20260812 09:45:07.986222 13902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:45:07.986515 13902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:45:07.988647 13902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786527905902985-13708-0/minicluster-data/ts-0/data/instance:
uuid: "2b24380489fd4462b51f4f84b1676892"
format_stamp: "Formatted at 2026-08-12 09:45:07 on dist-test-slave-jq4t"
server_key: "ae9cb90debd8db428de501be87bde972"
server_key_iv: "60397d62bc6a0662f9ca623f7082721a"
server_key_version: "encryptionkey@0"
I20260812 09:45:07.989176 13902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786527905902985-13708-0/minicluster-data/ts-0/wal/instance:
uuid: "2b24380489fd4462b51f4f84b1676892"
format_stamp: "Formatted at 2026-08-12 09:45:07 on dist-test-slave-jq4t"
server_key: "ae9cb90debd8db428de501be87bde972"
server_key_iv: "60397d62bc6a0662f9ca623f7082721a"
server_key_version: "encryptionkey@0"
I20260812 09:45:07.992771 13902 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260812 09:45:07.995142 13931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:07.996392 13902 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 09:45:07.996614 13902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786527905902985-13708-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786527905902985-13708-0/minicluster-data/ts-0/wal
uuid: "2b24380489fd4462b51f4f84b1676892"
format_stamp: "Formatted at 2026-08-12 09:45:07 on dist-test-slave-jq4t"
server_key: "ae9cb90debd8db428de501be87bde972"
server_key_iv: "60397d62bc6a0662f9ca623f7082721a"
server_key_version: "encryptionkey@0"
I20260812 09:45:07.996721 13902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786527905902985-13708-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786527905902985-13708-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786527905902985-13708-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:45:08.010632 13902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:45:08.013751 13902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:45:08.013974 13902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:45:08.014710 13902 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 09:45:08.015993 13902 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:45:08.016047 13902 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:08.016089 13902 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:45:08.016110 13902 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:08.025890 13902 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.1:38663
I20260812 09:45:08.025914 14108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.1:38663 every 8 connection(s)
I20260812 09:45:08.026765 13902 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786527905902985-13708-0/minicluster-data/ts-0/data/info.pb
I20260812 09:45:08.031453 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 13902
I20260812 09:45:08.031600 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786527905902985-13708-0/minicluster-data/ts-0/wal/instance
I20260812 09:45:08.031883 13708 external_mini_cluster.cc:1428] Setting key 84b69327c1f2f168a7cf2b94ad97c358
Aug 12 09:45:08 dist-test-slave-jq4t krb5kdc[13719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527907, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.1@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
I20260812 09:45:08.043300 13857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:08.029158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.1:35359 (local address 127.13.99.62:41365)
0812 09:45:08.029296 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:08.029301 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:08.030237 (+ 936us) server_negotiation.cc:407] Connection header received
0812 09:45:08.032197 (+ 1960us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:08.032210 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:08.032267 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:08.032347 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:08.033929 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:08.034459 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:08.036174 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:08.036600 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:08.036724 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:08.039253 (+ 2529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:08.039256 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:08.039259 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:08.039284 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:08.040944 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:08.041589 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:08.041591 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:08.041593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:08.041670 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:08.042100 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:08.042102 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:08.042103 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:08.042280 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:08.042356 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:08.043068 (+ 712us) server_negotiation.cc:299] Negotiation successful
0812 09:45:08.043172 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
WARNING: no policy specified for kudu/127.13.99.2@KRBTEST.COM; defaulting to no policy
I20260812 09:45:08.044286 14111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:08.029555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:41365 (local address 127.13.99.1:35359)
0812 09:45:08.030072 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:08.030116 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:08.031932 (+ 1816us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:08.032626 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:08.032636 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:08.033073 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:45:08.033759 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:08.033781 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:08.034609 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:08.034616 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:08.036037 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:08.036048 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:08.036748 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:08.036751 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:08.036901 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:08.037577 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:45:08.037595 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:45:08.039070 (+ 1475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:45:08.041077 (+ 2007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:08.041085 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:08.041102 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:08.041477 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:08.041765 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:08.041769 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:08.041773 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:08.041997 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:08.042401 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:45:08.042411 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:45:08.042774 (+ 363us) client_negotiation.cc:769] Sending connection context
0812 09:45:08.043034 (+ 260us) client_negotiation.cc:240] Negotiation successful
0812 09:45:08.043399 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":122,"threads_started":1}
I20260812 09:45:08.045722 14109 heartbeater.cc:347] Connected to a master server at 127.13.99.62:41365
I20260812 09:45:08.046000 14109 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.99.2@KRBTEST.COM" created.
I20260812 09:45:08.046648 14109 heartbeater.cc:511] Master 127.13.99.62:41365 requested a full tablet report, sending...
I20260812 09:45:08.049014 13796 ts_manager.cc:194] Registered new tserver with Master: 2b24380489fd4462b51f4f84b1676892 (127.13.99.1:38663)
I20260812 09:45:08.050570 13796 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.99.1@KRBTEST.COM'} at 127.13.99.1:35359
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.99.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.2.keytab.
Entry for principal kudu/127.13.99.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.2.keytab.
I20260812 09:45:08.063649 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786527905902985-13708-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786527905902985-13708-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786527905902985-13708-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786527905902985-13708-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.99.2:0
--local_ip_for_outbound_sockets=127.13.99.2
--webserver_interface=127.13.99.2
--webserver_port=0
--tserver_master_addrs=127.13.99.62:41365
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.2.keytab
--principal=kudu/127.13.99.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:42299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:45:08.178599 14116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:45:08.178872 14116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:45:08.178953 14116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:45:08.182875 14116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:45:08.182962 14116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:45:08.183054 14116 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.99.2
I20260812 09:45:08.188572 14116 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:42299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786527905902985-13708-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786527905902985-13708-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.2.keytab
--principal=kudu/127.13.99.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.99.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786527905902985-13708-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.99.2
--webserver_port=0
--tserver_master_addrs=127.13.99.62:41365
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14116
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.99.2
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786527905902985-13708-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:45:08.189604 14116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:45:08.190606 14116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:45:08.200695 14122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:08.200737 14124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:08.200706 14127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:45:08.201107 14116 server_base.cc:1034] running on GCE node
I20260812 09:45:08.201545 14116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:45:08.202226 14116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:45:08.203449 14116 hybrid_clock.cc:648] HybridClock initialized: now 1786527908203432 us; error 29 us; skew 500 ppm
Aug 12 09:45:08 dist-test-slave-jq4t krb5kdc[13719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527908, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:45:08.206532 14116 init.cc:377] Logged in from keytab as kudu/127.13.99.2@KRBTEST.COM (short username kudu)
I20260812 09:45:08.207508 14116 webserver.cc:460] Webserver started at http://127.13.99.2:35217/ using document root <none> and password file <none>
I20260812 09:45:08.208103 14116 fs_manager.cc:359] Metadata directory not provided
I20260812 09:45:08.208197 14116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:45:08.208441 14116 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:45:08.210556 14116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786527905902985-13708-0/minicluster-data/ts-1/data/instance:
uuid: "c19998cddd2742d29e04e0d85754e38b"
format_stamp: "Formatted at 2026-08-12 09:45:08 on dist-test-slave-jq4t"
server_key: "55219b08e2d45506a4a3720af2c823d5"
server_key_iv: "bcfd347e0cda0d6c919ab3a43f387508"
server_key_version: "encryptionkey@0"
I20260812 09:45:08.211138 14116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786527905902985-13708-0/minicluster-data/ts-1/wal/instance:
uuid: "c19998cddd2742d29e04e0d85754e38b"
format_stamp: "Formatted at 2026-08-12 09:45:08 on dist-test-slave-jq4t"
server_key: "55219b08e2d45506a4a3720af2c823d5"
server_key_iv: "bcfd347e0cda0d6c919ab3a43f387508"
server_key_version: "encryptionkey@0"
I20260812 09:45:08.215148 14116 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260812 09:45:08.217841 14139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:08.219092 14116 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260812 09:45:08.219229 14116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786527905902985-13708-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786527905902985-13708-0/minicluster-data/ts-1/wal
uuid: "c19998cddd2742d29e04e0d85754e38b"
format_stamp: "Formatted at 2026-08-12 09:45:08 on dist-test-slave-jq4t"
server_key: "55219b08e2d45506a4a3720af2c823d5"
server_key_iv: "bcfd347e0cda0d6c919ab3a43f387508"
server_key_version: "encryptionkey@0"
I20260812 09:45:08.219338 14116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786527905902985-13708-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786527905902985-13708-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786527905902985-13708-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:45:08.241919 14116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:45:08.244917 14116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:45:08.245129 14116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:45:08.245735 14116 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 09:45:08.246564 14116 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:45:08.246636 14116 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:08.246708 14116 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:45:08.246752 14116 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:08.256393 14116 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.2:37645
I20260812 09:45:08.256417 14324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.2:37645 every 8 connection(s)
I20260812 09:45:08.257323 14116 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786527905902985-13708-0/minicluster-data/ts-1/data/info.pb
I20260812 09:45:08.260600 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 14116
I20260812 09:45:08.260710 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786527905902985-13708-0/minicluster-data/ts-1/wal/instance
I20260812 09:45:08.261065 13708 external_mini_cluster.cc:1428] Setting key 7f0bb122c8fe7f2c8e895820d8e209ff
Aug 12 09:45:08 dist-test-slave-jq4t krb5kdc[13719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527908, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.2@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.99.3@KRBTEST.COM; defaulting to no policy
I20260812 09:45:08.272837 13857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:08.259365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.2:37561 (local address 127.13.99.62:41365)
0812 09:45:08.259502 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:08.259506 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:08.260385 (+ 879us) server_negotiation.cc:407] Connection header received
0812 09:45:08.262106 (+ 1721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:08.262110 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:08.262166 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:08.262255 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:08.263490 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:08.264035 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:08.265803 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:08.266078 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:08.266182 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:08.269159 (+ 2977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:08.269162 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:08.269164 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:08.269188 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:08.270696 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:08.271229 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:08.271237 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:08.271239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:08.271315 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:08.271671 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:08.271674 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:08.271676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:08.271909 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:08.272026 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:08.272540 (+ 514us) server_negotiation.cc:299] Negotiation successful
0812 09:45:08.272666 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260812 09:45:08.273604 14330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:08.259705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:41365 (local address 127.13.99.2:37561)
0812 09:45:08.260216 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:08.260258 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:08.261899 (+ 1641us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:08.262441 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:08.262450 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:08.262843 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:45:08.263338 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:08.263353 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:08.264169 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:08.264173 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:08.265666 (+ 1493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:08.265681 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:08.266249 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:08.266256 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:08.266502 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:08.267139 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:45:08.267165 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:45:08.268979 (+ 1814us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:45:08.270834 (+ 1855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:08.270840 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:08.270854 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:08.271115 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:08.271409 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:08.271413 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:08.271415 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:08.271565 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:08.272026 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:45:08.272034 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:45:08.272278 (+ 244us) client_negotiation.cc:769] Sending connection context
0812 09:45:08.272468 (+ 190us) client_negotiation.cc:240] Negotiation successful
0812 09:45:08.272804 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":142,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.99.3@KRBTEST.COM" created.
I20260812 09:45:08.274796 14327 heartbeater.cc:347] Connected to a master server at 127.13.99.62:41365
I20260812 09:45:08.275061 14327 heartbeater.cc:464] Registering TS with master...
I20260812 09:45:08.275674 14327 heartbeater.cc:511] Master 127.13.99.62:41365 requested a full tablet report, sending...
I20260812 09:45:08.276971 13796 ts_manager.cc:194] Registered new tserver with Master: c19998cddd2742d29e04e0d85754e38b (127.13.99.2:37645)
I20260812 09:45:08.277890 13796 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.99.2@KRBTEST.COM'} at 127.13.99.2:37561
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.99.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.3.keytab.
Entry for principal kudu/127.13.99.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.3.keytab.
I20260812 09:45:08.290880 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786527905902985-13708-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786527905902985-13708-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786527905902985-13708-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786527905902985-13708-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.99.3:0
--local_ip_for_outbound_sockets=127.13.99.3
--webserver_interface=127.13.99.3
--webserver_port=0
--tserver_master_addrs=127.13.99.62:41365
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.3.keytab
--principal=kudu/127.13.99.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:42299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:45:08.406142 14334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:45:08.406381 14334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:45:08.406423 14334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:45:08.410328 14334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:45:08.410377 14334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:45:08.410432 14334 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.99.3
I20260812 09:45:08.415771 14334 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:42299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786527905902985-13708-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786527905902985-13708-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.3.keytab
--principal=kudu/127.13.99.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.99.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786527905902985-13708-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.99.3
--webserver_port=0
--tserver_master_addrs=127.13.99.62:41365
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14334
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.99.3
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786527905902985-13708-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:45:08.416958 14334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:45:08.418032 14334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:45:08.427619 14344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:08.427613 14342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:08.427613 14341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:45:08.427708 14334 server_base.cc:1034] running on GCE node
I20260812 09:45:08.428277 14334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:45:08.428946 14334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:45:08.430157 14334 hybrid_clock.cc:648] HybridClock initialized: now 1786527908430148 us; error 32 us; skew 500 ppm
Aug 12 09:45:08 dist-test-slave-jq4t krb5kdc[13719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527908, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:45:08.432947 14334 init.cc:377] Logged in from keytab as kudu/127.13.99.3@KRBTEST.COM (short username kudu)
I20260812 09:45:08.433840 14334 webserver.cc:460] Webserver started at http://127.13.99.3:43819/ using document root <none> and password file <none>
I20260812 09:45:08.434397 14334 fs_manager.cc:359] Metadata directory not provided
I20260812 09:45:08.434501 14334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:45:08.434736 14334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:45:08.436826 14334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786527905902985-13708-0/minicluster-data/ts-2/data/instance:
uuid: "8c6c2aeb65a149bb9d9d382d2ceb0386"
format_stamp: "Formatted at 2026-08-12 09:45:08 on dist-test-slave-jq4t"
server_key: "a12e02a89a97fd444646bbd653cbea31"
server_key_iv: "e8c0bca84ff156f8f1b106cc69547dad"
server_key_version: "encryptionkey@0"
I20260812 09:45:08.437407 14334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786527905902985-13708-0/minicluster-data/ts-2/wal/instance:
uuid: "8c6c2aeb65a149bb9d9d382d2ceb0386"
format_stamp: "Formatted at 2026-08-12 09:45:08 on dist-test-slave-jq4t"
server_key: "a12e02a89a97fd444646bbd653cbea31"
server_key_iv: "e8c0bca84ff156f8f1b106cc69547dad"
server_key_version: "encryptionkey@0"
I20260812 09:45:08.441030 14334 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260812 09:45:08.443440 14356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:08.444770 14334 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 09:45:08.444905 14334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786527905902985-13708-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786527905902985-13708-0/minicluster-data/ts-2/wal
uuid: "8c6c2aeb65a149bb9d9d382d2ceb0386"
format_stamp: "Formatted at 2026-08-12 09:45:08 on dist-test-slave-jq4t"
server_key: "a12e02a89a97fd444646bbd653cbea31"
server_key_iv: "e8c0bca84ff156f8f1b106cc69547dad"
server_key_version: "encryptionkey@0"
I20260812 09:45:08.445021 14334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786527905902985-13708-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786527905902985-13708-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786527905902985-13708-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:45:08.471797 14334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:45:08.474946 14334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:45:08.475154 14334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:45:08.475754 14334 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 09:45:08.476609 14334 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:45:08.476691 14334 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:08.476764 14334 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:45:08.476817 14334 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:08.486024 14334 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.3:45049
I20260812 09:45:08.486063 14542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.3:45049 every 8 connection(s)
I20260812 09:45:08.486930 14334 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786527905902985-13708-0/minicluster-data/ts-2/data/info.pb
I20260812 09:45:08.487723 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 14334
I20260812 09:45:08.487849 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786527905902985-13708-0/minicluster-data/ts-2/wal/instance
I20260812 09:45:08.488135 13708 external_mini_cluster.cc:1428] Setting key 8b042882b0bdd76e6c6c91fc79e1c01b
Aug 12 09:45:08 dist-test-slave-jq4t krb5kdc[13719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527908, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.3@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
I20260812 09:45:08.501204 13857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:08.488954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.3:40813 (local address 127.13.99.62:41365)
0812 09:45:08.489119 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:08.489124 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:08.490195 (+ 1071us) server_negotiation.cc:407] Connection header received
0812 09:45:08.491577 (+ 1382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:08.491580 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:08.491636 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:08.491715 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:08.493072 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:08.493620 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:08.494785 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:08.495013 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:08.495127 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:08.497607 (+ 2480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:08.497611 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:08.497613 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:08.497644 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:08.498978 (+ 1334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:08.499510 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:08.499515 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:08.499517 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:08.499621 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:08.499982 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:08.499986 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:08.499988 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:08.500232 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:08.500349 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:08.500909 (+ 560us) server_negotiation.cc:299] Negotiation successful
0812 09:45:08.501032 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260812 09:45:08.502156 14547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:08.489377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:41365 (local address 127.13.99.3:40813)
0812 09:45:08.490054 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:08.490099 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:08.491368 (+ 1269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:08.491896 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:08.491903 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:08.492252 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:45:08.492923 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:08.492941 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:08.493749 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:08.493753 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:08.494651 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:08.494661 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:08.495164 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:08.495168 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:08.495340 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:08.495958 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:45:08.495975 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:45:08.497431 (+ 1456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:45:08.499112 (+ 1681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:08.499123 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:08.499137 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:08.499392 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:08.499716 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:08.499721 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:08.499726 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:08.499879 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:08.500347 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:45:08.500354 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:45:08.500655 (+ 301us) client_negotiation.cc:769] Sending connection context
0812 09:45:08.500851 (+ 196us) client_negotiation.cc:240] Negotiation successful
0812 09:45:08.501173 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":139,"threads_started":1}
I20260812 09:45:08.503309 14543 heartbeater.cc:347] Connected to a master server at 127.13.99.62:41365
I20260812 09:45:08.503561 14543 heartbeater.cc:464] Registering TS with master...
I20260812 09:45:08.504029 14543 heartbeater.cc:511] Master 127.13.99.62:41365 requested a full tablet report, sending...
I20260812 09:45:08.505115 13796 ts_manager.cc:194] Registered new tserver with Master: 8c6c2aeb65a149bb9d9d382d2ceb0386 (127.13.99.3:45049)
I20260812 09:45:08.505735 13796 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.99.3@KRBTEST.COM'} at 127.13.99.3:40813
I20260812 09:45:08.514992 13708 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 09:45:08 dist-test-slave-jq4t krb5kdc[13719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527908, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 09:45:08 dist-test-slave-jq4t krb5kdc[13719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527908, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.13.99.1@KRBTEST.COM
I20260812 09:45:08.543258 14561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:08.533778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53204 (local address 127.13.99.1:38663)
0812 09:45:08.534074 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:08.534099 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:08.534124 (+ 25us) server_negotiation.cc:407] Connection header received
0812 09:45:08.534176 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:08.534181 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:08.534374 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:08.534499 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:08.535159 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:08.536044 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:08.536928 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:08.537316 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:08.537402 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:08.539828 (+ 2426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:08.539837 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:08.539851 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:08.539874 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:08.541415 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:08.541875 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:08.541885 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:08.541887 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:08.541966 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:08.542270 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:08.542274 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:08.542276 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:08.542520 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:08.542737 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:08.542924 (+ 187us) server_negotiation.cc:299] Negotiation successful
0812 09:45:08.543089 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":109,"threads_started":1}
W20260812 09:45:08.544312 13972 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53204
Aug 12 09:45:08 dist-test-slave-jq4t krb5kdc[13719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 09:45:08 dist-test-slave-jq4t krb5kdc[13719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.99.1@KRBTEST.COM
I20260812 09:45:08.570449 14561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:08.562495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53212 (local address 127.13.99.1:38663)
0812 09:45:08.562646 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:08.562650 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:08.562704 (+ 54us) server_negotiation.cc:407] Connection header received
0812 09:45:08.562808 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:08.562810 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:08.562847 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:08.562914 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:08.563544 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:08.564052 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:08.564887 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:08.565125 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:08.565202 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:08.567384 (+ 2182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:08.567387 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:08.567389 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:08.567411 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:08.568955 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:08.569308 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:08.569310 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:08.569312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:08.569387 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:08.569682 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:08.569685 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:08.569686 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:08.569839 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:08.569957 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:08.570156 (+ 199us) server_negotiation.cc:299] Negotiation successful
0812 09:45:08.570252 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260812 09:45:08.571633 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 13902
I20260812 09:45:08.588531 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 14116
I20260812 09:45:08.611956 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 14334
I20260812 09:45:08.622100 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 13737
2026-08-12T09:45:08Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2685 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786527905902985-13708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 09:45:08 dist-test-slave-jq4t krb5kdc[14568](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 09:45:08 dist-test-slave-jq4t krb5kdc[14568](info): set up 2 sockets
Aug 12 09:45:08 dist-test-slave-jq4t krb5kdc[14568](info): commencing operation
krb5kdc: starting...
W20260812 09:45:09.901347 13708 mini_kdc.cc:121] Time spent starting KDC: real 1.248s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786527905902985-13708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786527905902985-13708-0/krb5kdc/test-user.keytab.
Aug 12 09:45:09 dist-test-slave-jq4t krb5kdc[14568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527909, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T09:45:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T09:45:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.99.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.99.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.99.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab.
Entry for principal kudu/127.13.99.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab.
I20260812 09:45:10.029737 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786527905902985-13708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786527905902985-13708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786527905902985-13708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.62:35481
--webserver_interface=127.13.99.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab
--principal=kudu/127.13.99.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:38153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.99.62:35481
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:45:10.142747 14595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:45:10.143002 14595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:45:10.143040 14595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:45:10.146908 14595 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 09:45:10.146955 14595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:45:10.146974 14595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:45:10.147037 14595 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 09:45:10.147063 14595 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 09:45:10.152184 14595 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:38153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786527905902985-13708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786527905902985-13708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.99.62:35481
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab
--principal=kudu/127.13.99.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.62:35481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.99.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14595
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786527905902985-13708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:45:10.153184 14595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:45:10.154163 14595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:45:10.162662 14611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:10.162657 14613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:10.162676 14610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:45:10.162729 14595 server_base.cc:1034] running on GCE node
I20260812 09:45:10.163517 14595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:45:10.164625 14595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:45:10.165795 14595 hybrid_clock.cc:648] HybridClock initialized: now 1786527910165776 us; error 31 us; skew 500 ppm
Aug 12 09:45:10 dist-test-slave-jq4t krb5kdc[14568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527910, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:45:10.168615 14595 init.cc:377] Logged in from keytab as kudu/127.13.99.62@KRBTEST.COM (short username kudu)
I20260812 09:45:10.169551 14595 webserver.cc:460] Webserver started at http://127.13.99.62:45189/ using document root <none> and password file <none>
I20260812 09:45:10.170132 14595 fs_manager.cc:359] Metadata directory not provided
I20260812 09:45:10.170197 14595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:45:10.170462 14595 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:45:10.172577 14595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786527905902985-13708-0/minicluster-data/master-0/data/instance:
uuid: "b2870e3dc1154d71984f5ea684a15dc1"
format_stamp: "Formatted at 2026-08-12 09:45:10 on dist-test-slave-jq4t"
server_key: "c3bf868ca66e056311001365b5aec9ff"
server_key_iv: "6774fc37298c59510c417de4559a659f"
server_key_version: "encryptionkey@0"
I20260812 09:45:10.173166 14595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786527905902985-13708-0/minicluster-data/master-0/wal/instance:
uuid: "b2870e3dc1154d71984f5ea684a15dc1"
format_stamp: "Formatted at 2026-08-12 09:45:10 on dist-test-slave-jq4t"
server_key: "c3bf868ca66e056311001365b5aec9ff"
server_key_iv: "6774fc37298c59510c417de4559a659f"
server_key_version: "encryptionkey@0"
I20260812 09:45:10.176854 14595 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260812 09:45:10.179186 14621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:10.180437 14595 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 09:45:10.180651 14595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786527905902985-13708-0/minicluster-data/master-0/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786527905902985-13708-0/minicluster-data/master-0/wal
uuid: "b2870e3dc1154d71984f5ea684a15dc1"
format_stamp: "Formatted at 2026-08-12 09:45:10 on dist-test-slave-jq4t"
server_key: "c3bf868ca66e056311001365b5aec9ff"
server_key_iv: "6774fc37298c59510c417de4559a659f"
server_key_version: "encryptionkey@0"
I20260812 09:45:10.180768 14595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786527905902985-13708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786527905902985-13708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786527905902985-13708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:45:10.194480 14595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:45:10.197455 14595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:45:10.197661 14595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:45:10.205204 14595 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.62:35481
I20260812 09:45:10.205232 14730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.62:35481 every 8 connection(s)
I20260812 09:45:10.206218 14595 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
I20260812 09:45:10.209152 14731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:45:10.214816 14731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2870e3dc1154d71984f5ea684a15dc1: Bootstrap starting.
I20260812 09:45:10.216230 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 14595
I20260812 09:45:10.216349 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786527905902985-13708-0/minicluster-data/master-0/wal/instance
I20260812 09:45:10.216655 13708 external_mini_cluster.cc:1428] Setting key e995aca68c442f493b2a394f9f84e3d5
I20260812 09:45:10.217824 14731 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2870e3dc1154d71984f5ea684a15dc1: Neither blocks nor log segments found. Creating new log.
I20260812 09:45:10.218675 14731 log.cc:824] T 00000000000000000000000000000000 P b2870e3dc1154d71984f5ea684a15dc1: Log is configured to *not* fsync() on all Append() calls
I20260812 09:45:10.220755 14731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2870e3dc1154d71984f5ea684a15dc1: No bootstrap required, opened a new log
I20260812 09:45:10.224102 14731 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2870e3dc1154d71984f5ea684a15dc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2870e3dc1154d71984f5ea684a15dc1" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 35481 } }
I20260812 09:45:10.224340 14731 raft_consensus.cc:374] T 00000000000000000000000000000000 P b2870e3dc1154d71984f5ea684a15dc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:45:10.224423 14731 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2870e3dc1154d71984f5ea684a15dc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2870e3dc1154d71984f5ea684a15dc1, State: Initialized, Role: FOLLOWER
I20260812 09:45:10.224886 14731 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2870e3dc1154d71984f5ea684a15dc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2870e3dc1154d71984f5ea684a15dc1" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 35481 } }
I20260812 09:45:10.225046 14731 raft_consensus.cc:388] T 00000000000000000000000000000000 P b2870e3dc1154d71984f5ea684a15dc1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 09:45:10.225137 14731 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2870e3dc1154d71984f5ea684a15dc1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 09:45:10.225258 14731 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2870e3dc1154d71984f5ea684a15dc1 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:45:10.226157 14731 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2870e3dc1154d71984f5ea684a15dc1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2870e3dc1154d71984f5ea684a15dc1" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 35481 } }
I20260812 09:45:10.226509 14731 leader_election.cc:302] T 00000000000000000000000000000000 P b2870e3dc1154d71984f5ea684a15dc1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b2870e3dc1154d71984f5ea684a15dc1; no voters:
I20260812 09:45:10.226828 14731 leader_election.cc:288] T 00000000000000000000000000000000 P b2870e3dc1154d71984f5ea684a15dc1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 09:45:10.227103 14736 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b2870e3dc1154d71984f5ea684a15dc1 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 09:45:10.227444 14736 raft_consensus.cc:686] T 00000000000000000000000000000000 P b2870e3dc1154d71984f5ea684a15dc1 [term 1 LEADER]: Becoming Leader. State: Replica: b2870e3dc1154d71984f5ea684a15dc1, State: Running, Role: LEADER
I20260812 09:45:10.227831 14736 consensus_queue.cc:237] T 00000000000000000000000000000000 P b2870e3dc1154d71984f5ea684a15dc1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2870e3dc1154d71984f5ea684a15dc1" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 35481 } }
I20260812 09:45:10.228215 14731 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2870e3dc1154d71984f5ea684a15dc1 [sys.catalog]: configured and running, proceeding with master startup.
Aug 12 09:45:10 dist-test-slave-jq4t krb5kdc[14568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527909, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
I20260812 09:45:10.229682 14737 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2870e3dc1154d71984f5ea684a15dc1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b2870e3dc1154d71984f5ea684a15dc1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2870e3dc1154d71984f5ea684a15dc1" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 35481 } } }
I20260812 09:45:10.229818 14737 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2870e3dc1154d71984f5ea684a15dc1 [sys.catalog]: This master's current role is: LEADER
I20260812 09:45:10.230464 14740 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2870e3dc1154d71984f5ea684a15dc1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2870e3dc1154d71984f5ea684a15dc1. Latest consensus state: current_term: 1 leader_uuid: "b2870e3dc1154d71984f5ea684a15dc1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2870e3dc1154d71984f5ea684a15dc1" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 35481 } } }
I20260812 09:45:10.230578 14740 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2870e3dc1154d71984f5ea684a15dc1 [sys.catalog]: This master's current role is: LEADER
W20260812 09:45:10.233785 14761 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b2870e3dc1154d71984f5ea684a15dc1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 09:45:10.233865 14761 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 09:45:10.233978 14751 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 09:45:10.234793 14734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:10.217942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36104 (local address 127.13.99.62:35481)
0812 09:45:10.218447 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:10.218473 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:10.218511 (+ 38us) server_negotiation.cc:407] Connection header received
0812 09:45:10.219406 (+ 895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:10.219436 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:10.219807 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:10.220155 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:10.221076 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:10.222091 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:10.222977 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:10.223459 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:10.223638 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:10.229777 (+ 6139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:10.229784 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:10.229802 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:10.229834 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:10.231899 (+ 2065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:10.232376 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:10.232382 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:10.232387 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:10.232565 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:10.232862 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:10.232866 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:10.232867 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:10.233260 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:10.233481 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:10.233724 (+ 243us) server_negotiation.cc:299] Negotiation successful
0812 09:45:10.234063 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":134,"threads_started":1}
I20260812 09:45:10.234812 14751 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 09:45:10.240631 14751 catalog_manager.cc:1347] Generated new cluster ID: 484202ac4bc54918b394190bd4b12ce3
I20260812 09:45:10.240705 14751 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 09:45:10.256646 14751 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 09:45:10.257573 14751 catalog_manager.cc:1504] Loading token signing keys...
I20260812 09:45:10.263780 14751 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b2870e3dc1154d71984f5ea684a15dc1: Generated new TSK 0
I20260812 09:45:10.264461 14751 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.99.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.99.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.99.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.1.keytab.
Entry for principal kudu/127.13.99.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.1.keytab.
I20260812 09:45:10.296200 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786527905902985-13708-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786527905902985-13708-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786527905902985-13708-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786527905902985-13708-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.99.1:0
--local_ip_for_outbound_sockets=127.13.99.1
--webserver_interface=127.13.99.1
--webserver_port=0
--tserver_master_addrs=127.13.99.62:35481
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.1.keytab
--principal=kudu/127.13.99.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:38153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:45:10.412854 14769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:45:10.413126 14769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:45:10.413172 14769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:45:10.417207 14769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:45:10.417268 14769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:45:10.417326 14769 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.99.1
I20260812 09:45:10.422618 14769 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:38153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786527905902985-13708-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786527905902985-13708-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.1.keytab
--principal=kudu/127.13.99.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.99.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786527905902985-13708-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.99.1
--webserver_port=0
--tserver_master_addrs=127.13.99.62:35481
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14769
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.99.1
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786527905902985-13708-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:45:10.423597 14769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:45:10.424590 14769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:45:10.434664 14781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:10.434687 14785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:45:10.434762 14769 server_base.cc:1034] running on GCE node
W20260812 09:45:10.434715 14777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:45:10.435288 14769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:45:10.435925 14769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:45:10.437139 14769 hybrid_clock.cc:648] HybridClock initialized: now 1786527910437113 us; error 27 us; skew 500 ppm
Aug 12 09:45:10 dist-test-slave-jq4t krb5kdc[14568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527910, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:45:10.439890 14769 init.cc:377] Logged in from keytab as kudu/127.13.99.1@KRBTEST.COM (short username kudu)
I20260812 09:45:10.440827 14769 webserver.cc:460] Webserver started at http://127.13.99.1:39287/ using document root <none> and password file <none>
I20260812 09:45:10.441409 14769 fs_manager.cc:359] Metadata directory not provided
I20260812 09:45:10.441502 14769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:45:10.441740 14769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:45:10.443770 14769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786527905902985-13708-0/minicluster-data/ts-0/data/instance:
uuid: "1d9f2d3f220e4cbc9bee179bd669d07f"
format_stamp: "Formatted at 2026-08-12 09:45:10 on dist-test-slave-jq4t"
server_key: "b9e0ad9cffd86a2729ee9561db8551f6"
server_key_iv: "f3de02eef8b29ab9ce04514469a5489f"
server_key_version: "encryptionkey@0"
I20260812 09:45:10.444341 14769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786527905902985-13708-0/minicluster-data/ts-0/wal/instance:
uuid: "1d9f2d3f220e4cbc9bee179bd669d07f"
format_stamp: "Formatted at 2026-08-12 09:45:10 on dist-test-slave-jq4t"
server_key: "b9e0ad9cffd86a2729ee9561db8551f6"
server_key_iv: "f3de02eef8b29ab9ce04514469a5489f"
server_key_version: "encryptionkey@0"
I20260812 09:45:10.448066 14769 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260812 09:45:10.450539 14792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:10.451741 14769 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 09:45:10.451879 14769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786527905902985-13708-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786527905902985-13708-0/minicluster-data/ts-0/wal
uuid: "1d9f2d3f220e4cbc9bee179bd669d07f"
format_stamp: "Formatted at 2026-08-12 09:45:10 on dist-test-slave-jq4t"
server_key: "b9e0ad9cffd86a2729ee9561db8551f6"
server_key_iv: "f3de02eef8b29ab9ce04514469a5489f"
server_key_version: "encryptionkey@0"
I20260812 09:45:10.452004 14769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786527905902985-13708-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786527905902985-13708-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786527905902985-13708-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:45:10.468811 14769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:45:10.471918 14769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:45:10.472126 14769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:45:10.472771 14769 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 09:45:10.473611 14769 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:45:10.473682 14769 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:10.473749 14769 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:45:10.473799 14769 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:10.482523 14769 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.1:45119
I20260812 09:45:10.482546 14992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.1:45119 every 8 connection(s)
I20260812 09:45:10.483428 14769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786527905902985-13708-0/minicluster-data/ts-0/data/info.pb
Aug 12 09:45:10 dist-test-slave-jq4t krb5kdc[14568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527910, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.1@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
I20260812 09:45:10.493034 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 14769
I20260812 09:45:10.493150 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786527905902985-13708-0/minicluster-data/ts-0/wal/instance
I20260812 09:45:10.493458 13708 external_mini_cluster.cc:1428] Setting key 93ca87b6d5f2400d03c4bf4bf1af7bdc
I20260812 09:45:10.497258 14734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:10.485433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.1:59887 (local address 127.13.99.62:35481)
0812 09:45:10.485590 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:10.485594 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:10.486406 (+ 812us) server_negotiation.cc:407] Connection header received
0812 09:45:10.487604 (+ 1198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:10.487607 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:10.487658 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:10.487732 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:10.488929 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:10.489482 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:10.490586 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:10.490845 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:10.490929 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:10.493559 (+ 2630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:10.493565 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:10.493569 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:10.493602 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:10.495124 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:10.495643 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:10.495647 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:10.495649 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:10.495735 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:10.496094 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:10.496098 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:10.496099 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:10.496285 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:10.496400 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:10.496953 (+ 553us) server_negotiation.cc:299] Negotiation successful
0812 09:45:10.497085 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260812 09:45:10.498039 14998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:10.485801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:35481 (local address 127.13.99.1:59887)
0812 09:45:10.486248 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:10.486294 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:10.487403 (+ 1109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:10.487902 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:10.487908 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:10.488241 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:45:10.488787 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:10.488801 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:10.489599 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:10.489602 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:10.490466 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:10.490475 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:10.490981 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:10.490984 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:10.491118 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:10.491708 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:45:10.491723 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:45:10.493357 (+ 1634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:45:10.495252 (+ 1895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:10.495259 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:10.495273 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:10.495534 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:10.495841 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:10.495844 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:10.495846 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:10.495988 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:10.496400 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:45:10.496408 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:45:10.496718 (+ 310us) client_negotiation.cc:769] Sending connection context
0812 09:45:10.496910 (+ 192us) client_negotiation.cc:240] Negotiation successful
0812 09:45:10.497230 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":118,"threads_started":1}
I20260812 09:45:10.499347 14995 heartbeater.cc:347] Connected to a master server at 127.13.99.62:35481
I20260812 09:45:10.499607 14995 heartbeater.cc:464] Registering TS with master...
I20260812 09:45:10.500169 14995 heartbeater.cc:511] Master 127.13.99.62:35481 requested a full tablet report, sending...
I20260812 09:45:10.502002 14654 ts_manager.cc:194] Registered new tserver with Master: 1d9f2d3f220e4cbc9bee179bd669d07f (127.13.99.1:45119)
I20260812 09:45:10.503692 14654 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.99.1@KRBTEST.COM'} at 127.13.99.1:59887
WARNING: no policy specified for kudu/127.13.99.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.99.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.99.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.2.keytab.
Entry for principal kudu/127.13.99.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.2.keytab.
I20260812 09:45:10.523790 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786527905902985-13708-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786527905902985-13708-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786527905902985-13708-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786527905902985-13708-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.99.2:0
--local_ip_for_outbound_sockets=127.13.99.2
--webserver_interface=127.13.99.2
--webserver_port=0
--tserver_master_addrs=127.13.99.62:35481
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.2.keytab
--principal=kudu/127.13.99.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:38153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:45:10.637396 15001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:45:10.637636 15001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:45:10.637678 15001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:45:10.641690 15001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:45:10.641739 15001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:45:10.641840 15001 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.99.2
I20260812 09:45:10.646997 15001 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:38153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786527905902985-13708-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786527905902985-13708-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.2.keytab
--principal=kudu/127.13.99.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.99.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786527905902985-13708-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.99.2
--webserver_port=0
--tserver_master_addrs=127.13.99.62:35481
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15001
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.99.2
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786527905902985-13708-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:45:10.648036 15001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:45:10.649124 15001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:45:10.659104 15015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:10.659114 15013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:10.659114 15012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:45:10.659335 15001 server_base.cc:1034] running on GCE node
I20260812 09:45:10.659861 15001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:45:10.660422 15001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:45:10.661631 15001 hybrid_clock.cc:648] HybridClock initialized: now 1786527910661616 us; error 26 us; skew 500 ppm
Aug 12 09:45:10 dist-test-slave-jq4t krb5kdc[14568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527910, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:45:10.664288 15001 init.cc:377] Logged in from keytab as kudu/127.13.99.2@KRBTEST.COM (short username kudu)
I20260812 09:45:10.665227 15001 webserver.cc:460] Webserver started at http://127.13.99.2:46845/ using document root <none> and password file <none>
I20260812 09:45:10.665791 15001 fs_manager.cc:359] Metadata directory not provided
I20260812 09:45:10.665885 15001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:45:10.666100 15001 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:45:10.668085 15001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786527905902985-13708-0/minicluster-data/ts-1/data/instance:
uuid: "09562dcffc8540e39c63a7632649b6d6"
format_stamp: "Formatted at 2026-08-12 09:45:10 on dist-test-slave-jq4t"
server_key: "255e3e04f32b3768e2249a6df32e7fe8"
server_key_iv: "758ad20f54283085fbe02c4774b2a77e"
server_key_version: "encryptionkey@0"
I20260812 09:45:10.668785 15001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786527905902985-13708-0/minicluster-data/ts-1/wal/instance:
uuid: "09562dcffc8540e39c63a7632649b6d6"
format_stamp: "Formatted at 2026-08-12 09:45:10 on dist-test-slave-jq4t"
server_key: "255e3e04f32b3768e2249a6df32e7fe8"
server_key_iv: "758ad20f54283085fbe02c4774b2a77e"
server_key_version: "encryptionkey@0"
I20260812 09:45:10.672420 15001 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260812 09:45:10.674774 15025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:10.675973 15001 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260812 09:45:10.676124 15001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786527905902985-13708-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786527905902985-13708-0/minicluster-data/ts-1/wal
uuid: "09562dcffc8540e39c63a7632649b6d6"
format_stamp: "Formatted at 2026-08-12 09:45:10 on dist-test-slave-jq4t"
server_key: "255e3e04f32b3768e2249a6df32e7fe8"
server_key_iv: "758ad20f54283085fbe02c4774b2a77e"
server_key_version: "encryptionkey@0"
I20260812 09:45:10.676237 15001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786527905902985-13708-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786527905902985-13708-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786527905902985-13708-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:45:10.691332 15001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:45:10.694396 15001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:45:10.694609 15001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:45:10.695179 15001 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 09:45:10.695986 15001 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:45:10.696058 15001 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:10.696126 15001 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:45:10.696180 15001 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:10.705256 15001 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.2:33565
I20260812 09:45:10.705296 15224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.2:33565 every 8 connection(s)
I20260812 09:45:10.706133 15001 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786527905902985-13708-0/minicluster-data/ts-1/data/info.pb
I20260812 09:45:10.710390 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 15001
I20260812 09:45:10.710484 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786527905902985-13708-0/minicluster-data/ts-1/wal/instance
I20260812 09:45:10.710762 13708 external_mini_cluster.cc:1428] Setting key 0f74142ed9011d42c80eb047d90455c2
Aug 12 09:45:10 dist-test-slave-jq4t krb5kdc[14568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527910, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.2@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
I20260812 09:45:10.720773 14734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:10.707835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.2:33557 (local address 127.13.99.62:35481)
0812 09:45:10.707982 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:10.707986 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:10.708826 (+ 840us) server_negotiation.cc:407] Connection header received
0812 09:45:10.710099 (+ 1273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:10.710102 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:10.710161 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:10.710264 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:10.711667 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:10.712307 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:10.713535 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:10.713755 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:10.713871 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:10.716812 (+ 2941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:10.716820 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:10.716826 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:10.716861 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:10.718551 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:10.719125 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:10.719129 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:10.719131 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:10.719224 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:10.719603 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:10.719606 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:10.719608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:10.719811 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:10.719928 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:10.720448 (+ 520us) server_negotiation.cc:299] Negotiation successful
0812 09:45:10.720596 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":15,"server-negotiator.queue_time_us":53}
I20260812 09:45:10.721726 15228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:10.708128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:35481 (local address 127.13.99.2:33557)
0812 09:45:10.708678 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:10.708719 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:10.709885 (+ 1166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:10.710434 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:10.710461 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:10.710938 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:45:10.711521 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:10.711540 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:10.712434 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:10.712438 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:10.713394 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:10.713404 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:10.713922 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:10.713927 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:10.714107 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:10.714789 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:45:10.714813 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:45:10.716634 (+ 1821us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:45:10.718688 (+ 2054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:10.718693 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:10.718707 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:10.718982 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:10.719327 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:10.719332 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:10.719337 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:10.719488 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:10.719926 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:45:10.719933 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:45:10.720189 (+ 256us) client_negotiation.cc:769] Sending connection context
0812 09:45:10.720384 (+ 195us) client_negotiation.cc:240] Negotiation successful
0812 09:45:10.720751 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":128,"threads_started":1}
WARNING: no policy specified for kudu/127.13.99.3@KRBTEST.COM; defaulting to no policy
I20260812 09:45:10.723017 15225 heartbeater.cc:347] Connected to a master server at 127.13.99.62:35481
I20260812 09:45:10.723315 15225 heartbeater.cc:464] Registering TS with master...
I20260812 09:45:10.723830 15225 heartbeater.cc:511] Master 127.13.99.62:35481 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.99.3@KRBTEST.COM" created.
I20260812 09:45:10.725152 14654 ts_manager.cc:194] Registered new tserver with Master: 09562dcffc8540e39c63a7632649b6d6 (127.13.99.2:33565)
I20260812 09:45:10.725996 14654 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.99.2@KRBTEST.COM'} at 127.13.99.2:33557
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.99.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.3.keytab.
Entry for principal kudu/127.13.99.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.3.keytab.
I20260812 09:45:10.744272 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786527905902985-13708-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786527905902985-13708-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786527905902985-13708-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786527905902985-13708-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.99.3:0
--local_ip_for_outbound_sockets=127.13.99.3
--webserver_interface=127.13.99.3
--webserver_port=0
--tserver_master_addrs=127.13.99.62:35481
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.3.keytab
--principal=kudu/127.13.99.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:38153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:45:10.858460 15231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:45:10.858693 15231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:45:10.858736 15231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:45:10.862668 15231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:45:10.862726 15231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:45:10.862784 15231 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.99.3
I20260812 09:45:10.867882 15231 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:38153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786527905902985-13708-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786527905902985-13708-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.3.keytab
--principal=kudu/127.13.99.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.99.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786527905902985-13708-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.99.3
--webserver_port=0
--tserver_master_addrs=127.13.99.62:35481
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15231
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.99.3
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786527905902985-13708-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:45:10.868932 15231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:45:10.869989 15231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:45:10.879653 15240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:10.879681 15242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:10.879645 15246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:45:10.879989 15231 server_base.cc:1034] running on GCE node
I20260812 09:45:10.880450 15231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:45:10.881105 15231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:45:10.882293 15231 hybrid_clock.cc:648] HybridClock initialized: now 1786527910882278 us; error 26 us; skew 500 ppm
Aug 12 09:45:10 dist-test-slave-jq4t krb5kdc[14568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527910, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:45:10.885157 15231 init.cc:377] Logged in from keytab as kudu/127.13.99.3@KRBTEST.COM (short username kudu)
I20260812 09:45:10.886056 15231 webserver.cc:460] Webserver started at http://127.13.99.3:40915/ using document root <none> and password file <none>
I20260812 09:45:10.886627 15231 fs_manager.cc:359] Metadata directory not provided
I20260812 09:45:10.886724 15231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:45:10.886962 15231 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:45:10.889056 15231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786527905902985-13708-0/minicluster-data/ts-2/data/instance:
uuid: "f951deec42b44fa88c83cb3aae72b7b6"
format_stamp: "Formatted at 2026-08-12 09:45:10 on dist-test-slave-jq4t"
server_key: "ea0b17fe292f5ef9f4bfd109bfd3f1f6"
server_key_iv: "5e009e66e7c44d911d978befdc5db643"
server_key_version: "encryptionkey@0"
I20260812 09:45:10.889635 15231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786527905902985-13708-0/minicluster-data/ts-2/wal/instance:
uuid: "f951deec42b44fa88c83cb3aae72b7b6"
format_stamp: "Formatted at 2026-08-12 09:45:10 on dist-test-slave-jq4t"
server_key: "ea0b17fe292f5ef9f4bfd109bfd3f1f6"
server_key_iv: "5e009e66e7c44d911d978befdc5db643"
server_key_version: "encryptionkey@0"
I20260812 09:45:10.893337 15231 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260812 09:45:10.895838 15255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:10.897179 15231 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260812 09:45:10.897324 15231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786527905902985-13708-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786527905902985-13708-0/minicluster-data/ts-2/wal
uuid: "f951deec42b44fa88c83cb3aae72b7b6"
format_stamp: "Formatted at 2026-08-12 09:45:10 on dist-test-slave-jq4t"
server_key: "ea0b17fe292f5ef9f4bfd109bfd3f1f6"
server_key_iv: "5e009e66e7c44d911d978befdc5db643"
server_key_version: "encryptionkey@0"
I20260812 09:45:10.897439 15231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786527905902985-13708-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786527905902985-13708-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786527905902985-13708-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:45:10.918670 15231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:45:10.922115 15231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:45:10.922343 15231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:45:10.922998 15231 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 09:45:10.924033 15231 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:45:10.924124 15231 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:10.924206 15231 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:45:10.924252 15231 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:10.934404 15231 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.3:35729
I20260812 09:45:10.934423 15454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.3:35729 every 8 connection(s)
I20260812 09:45:10.935355 15231 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786527905902985-13708-0/minicluster-data/ts-2/data/info.pb
I20260812 09:45:10.941313 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 15231
I20260812 09:45:10.941474 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786527905902985-13708-0/minicluster-data/ts-2/wal/instance
I20260812 09:45:10.941748 13708 external_mini_cluster.cc:1428] Setting key c0213dd4030574d3de95fb2395f9dbdc
Aug 12 09:45:10 dist-test-slave-jq4t krb5kdc[14568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527910, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.3@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
I20260812 09:45:10.949165 14734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:10.937351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.3:55557 (local address 127.13.99.62:35481)
0812 09:45:10.937537 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:10.937543 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:10.938369 (+ 826us) server_negotiation.cc:407] Connection header received
0812 09:45:10.939633 (+ 1264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:10.939636 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:10.939687 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:10.939764 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:10.940975 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:10.941623 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:10.943060 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:10.943386 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:10.943498 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:10.945953 (+ 2455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:10.945956 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:10.945958 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:10.945982 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:10.947430 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:10.947872 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:10.947875 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:10.947876 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:10.947947 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:10.948237 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:10.948239 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:10.948240 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:10.948404 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:10.948561 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:10.948948 (+ 387us) server_negotiation.cc:299] Negotiation successful
0812 09:45:10.949046 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260812 09:45:10.950178 15458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:10.937732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:35481 (local address 127.13.99.3:55557)
0812 09:45:10.938205 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:10.938250 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:10.939424 (+ 1174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:10.939932 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:10.939940 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:10.940321 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:45:10.940843 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:10.940857 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:10.941761 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:10.941765 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:10.942929 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:10.942939 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:10.943534 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:10.943537 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:10.943685 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:10.944308 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:45:10.944323 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:45:10.945805 (+ 1482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:45:10.947536 (+ 1731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:10.947540 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:10.947552 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:10.947787 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:10.948029 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:10.948031 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:10.948033 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:10.948157 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:10.948518 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:45:10.948523 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:45:10.948739 (+ 216us) client_negotiation.cc:769] Sending connection context
0812 09:45:10.948899 (+ 160us) client_negotiation.cc:240] Negotiation successful
0812 09:45:10.949146 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":122,"threads_started":1}
I20260812 09:45:10.951305 15456 heartbeater.cc:347] Connected to a master server at 127.13.99.62:35481
I20260812 09:45:10.951542 15456 heartbeater.cc:464] Registering TS with master...
I20260812 09:45:10.951992 15456 heartbeater.cc:511] Master 127.13.99.62:35481 requested a full tablet report, sending...
I20260812 09:45:10.953094 14654 ts_manager.cc:194] Registered new tserver with Master: f951deec42b44fa88c83cb3aae72b7b6 (127.13.99.3:35729)
I20260812 09:45:10.953665 14654 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.99.3@KRBTEST.COM'} at 127.13.99.3:55557
I20260812 09:45:10.956048 13708 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 09:45:10.967466 14734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:10.960400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36106 (local address 127.13.99.62:35481)
0812 09:45:10.960580 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:10.960584 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:10.960746 (+ 162us) server_negotiation.cc:407] Connection header received
0812 09:45:10.960840 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:10.960842 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:10.960882 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:10.960950 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:10.961644 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:10.962162 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:10.962993 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:10.963221 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:10.963310 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:10.964745 (+ 1435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:10.964748 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:10.964750 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:10.964770 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:10.966001 (+ 1231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:10.966402 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:10.966405 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:10.966406 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:10.966477 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:10.966722 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:10.966729 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:10.966730 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:10.966960 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:10.967060 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:10.967262 (+ 202us) server_negotiation.cc:299] Negotiation successful
0812 09:45:10.967351 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260812 09:45:10.973688 14654 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36106:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 09:45:10.977115 14654 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 09:45:10.993988 15475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:10.987473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.1:45119 (local address 127.0.0.1:50014)
0812 09:45:10.988278 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:10.988328 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:10.988471 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:10.989307 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:10.989313 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:10.989352 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:45:10.989622 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:10.989631 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:10.991181 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:10.991184 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:10.992448 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:10.992462 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:10.993385 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:10.993388 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:10.993512 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:10.993552 (+ 40us) client_negotiation.cc:769] Sending connection context
0812 09:45:10.993697 (+ 145us) client_negotiation.cc:240] Negotiation successful
0812 09:45:10.993830 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"mutex_wait_us":116,"spinlock_wait_cycles":14336,"thread_start_us":90,"threads_started":1}
I20260812 09:45:10.993990 15479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:10.987585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50014 (local address 127.13.99.1:45119)
0812 09:45:10.988333 (+ 748us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:10.988353 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:10.988646 (+ 293us) server_negotiation.cc:407] Connection header received
0812 09:45:10.988740 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:10.988745 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:10.988948 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:10.989112 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:45:10.989765 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:10.991037 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:10.992640 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:10.993244 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:10.993386 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:10.993570 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:10.993732 (+ 162us) server_negotiation.cc:299] Negotiation successful
0812 09:45:10.993830 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":645,"thread_start_us":114,"threads_started":1}
I20260812 09:45:10.994251 15477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:10.987560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.3:35729 (local address 127.0.0.1:43254)
0812 09:45:10.988471 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:10.988506 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:10.988606 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:10.989307 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:10.989313 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:10.989351 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:45:10.989622 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:10.989631 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:10.990835 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:10.990839 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:10.992358 (+ 1519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:10.992370 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:10.993739 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:10.993743 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:10.993836 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:10.993905 (+ 69us) client_negotiation.cc:769] Sending connection context
0812 09:45:10.993980 (+ 75us) client_negotiation.cc:240] Negotiation successful
0812 09:45:10.994082 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":857,"mutex_wait_us":231,"spinlock_wait_cycles":5760,"thread_start_us":116,"threads_started":1}
I20260812 09:45:10.994781 15482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:10.988659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36742 (local address 127.13.99.2:33565)
0812 09:45:10.989098 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:10.989113 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:10.989386 (+ 273us) server_negotiation.cc:407] Connection header received
0812 09:45:10.989535 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:10.989540 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:10.989708 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:10.989886 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:45:10.990096 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:10.991205 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:10.992750 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:10.993518 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:10.993628 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:10.993744 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:10.994517 (+ 773us) server_negotiation.cc:299] Negotiation successful
0812 09:45:10.994633 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":129,"threads_started":1}
I20260812 09:45:10.994804 15480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:10.987776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43254 (local address 127.13.99.3:35729)
0812 09:45:10.988371 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:10.988384 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:10.988703 (+ 319us) server_negotiation.cc:407] Connection header received
0812 09:45:10.988778 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:10.988782 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:10.988948 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:10.989104 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:45:10.989794 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:10.990682 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:10.992523 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:10.993392 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:10.993505 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:10.993670 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:10.994563 (+ 893us) server_negotiation.cc:299] Negotiation successful
0812 09:45:10.994655 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":131,"threads_started":1}
I20260812 09:45:10.995155 15481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:10.988235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.2:33565 (local address 127.0.0.1:36742)
0812 09:45:10.989258 (+ 1023us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:10.989273 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:10.989417 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:10.989871 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:10.989874 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:10.989893 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:45:10.989986 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:10.989991 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:10.991361 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:10.991366 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:10.992629 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:10.992638 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:10.993794 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:10.993797 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:10.993905 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:10.993924 (+ 19us) client_negotiation.cc:769] Sending connection context
0812 09:45:10.993986 (+ 62us) client_negotiation.cc:240] Negotiation successful
0812 09:45:10.995010 (+ 1024us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":959,"mutex_wait_us":967,"thread_start_us":109,"threads_started":1}
I20260812 09:45:10.997282 14886 tablet_service.cc:1467] Processing CreateTablet for tablet d214197a770048aebdc4c61a29d9b83e (DEFAULT_TABLE table=test-table [id=74bd7c85087f4ddba55a52fd99b5a766]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 09:45:10.997550 15349 tablet_service.cc:1467] Processing CreateTablet for tablet d214197a770048aebdc4c61a29d9b83e (DEFAULT_TABLE table=test-table [id=74bd7c85087f4ddba55a52fd99b5a766]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 09:45:10.998662 15349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d214197a770048aebdc4c61a29d9b83e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:45:10.998775 14886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d214197a770048aebdc4c61a29d9b83e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:45:10.999154 15111 tablet_service.cc:1467] Processing CreateTablet for tablet d214197a770048aebdc4c61a29d9b83e (DEFAULT_TABLE table=test-table [id=74bd7c85087f4ddba55a52fd99b5a766]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 09:45:11.000147 15111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d214197a770048aebdc4c61a29d9b83e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:45:11.005649 15484 tablet_bootstrap.cc:492] T d214197a770048aebdc4c61a29d9b83e P f951deec42b44fa88c83cb3aae72b7b6: Bootstrap starting.
I20260812 09:45:11.006896 15485 tablet_bootstrap.cc:492] T d214197a770048aebdc4c61a29d9b83e P 1d9f2d3f220e4cbc9bee179bd669d07f: Bootstrap starting.
I20260812 09:45:11.007045 15486 tablet_bootstrap.cc:492] T d214197a770048aebdc4c61a29d9b83e P 09562dcffc8540e39c63a7632649b6d6: Bootstrap starting.
I20260812 09:45:11.008196 15484 tablet_bootstrap.cc:654] T d214197a770048aebdc4c61a29d9b83e P f951deec42b44fa88c83cb3aae72b7b6: Neither blocks nor log segments found. Creating new log.
I20260812 09:45:11.008947 15485 tablet_bootstrap.cc:654] T d214197a770048aebdc4c61a29d9b83e P 1d9f2d3f220e4cbc9bee179bd669d07f: Neither blocks nor log segments found. Creating new log.
I20260812 09:45:11.009285 15486 tablet_bootstrap.cc:654] T d214197a770048aebdc4c61a29d9b83e P 09562dcffc8540e39c63a7632649b6d6: Neither blocks nor log segments found. Creating new log.
I20260812 09:45:11.009825 15485 log.cc:824] T d214197a770048aebdc4c61a29d9b83e P 1d9f2d3f220e4cbc9bee179bd669d07f: Log is configured to *not* fsync() on all Append() calls
I20260812 09:45:11.009891 15484 log.cc:824] T d214197a770048aebdc4c61a29d9b83e P f951deec42b44fa88c83cb3aae72b7b6: Log is configured to *not* fsync() on all Append() calls
I20260812 09:45:11.010452 15486 log.cc:824] T d214197a770048aebdc4c61a29d9b83e P 09562dcffc8540e39c63a7632649b6d6: Log is configured to *not* fsync() on all Append() calls
I20260812 09:45:11.011981 15484 tablet_bootstrap.cc:492] T d214197a770048aebdc4c61a29d9b83e P f951deec42b44fa88c83cb3aae72b7b6: No bootstrap required, opened a new log
I20260812 09:45:11.011976 15485 tablet_bootstrap.cc:492] T d214197a770048aebdc4c61a29d9b83e P 1d9f2d3f220e4cbc9bee179bd669d07f: No bootstrap required, opened a new log
I20260812 09:45:11.012120 15486 tablet_bootstrap.cc:492] T d214197a770048aebdc4c61a29d9b83e P 09562dcffc8540e39c63a7632649b6d6: No bootstrap required, opened a new log
I20260812 09:45:11.012185 15484 ts_tablet_manager.cc:1397] T d214197a770048aebdc4c61a29d9b83e P f951deec42b44fa88c83cb3aae72b7b6: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20260812 09:45:11.012185 15485 ts_tablet_manager.cc:1397] T d214197a770048aebdc4c61a29d9b83e P 1d9f2d3f220e4cbc9bee179bd669d07f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260812 09:45:11.012329 15486 ts_tablet_manager.cc:1397] T d214197a770048aebdc4c61a29d9b83e P 09562dcffc8540e39c63a7632649b6d6: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260812 09:45:11.015202 15486 raft_consensus.cc:348] T d214197a770048aebdc4c61a29d9b83e P 09562dcffc8540e39c63a7632649b6d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d9f2d3f220e4cbc9bee179bd669d07f" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 45119 } } peers { permanent_uuid: "f951deec42b44fa88c83cb3aae72b7b6" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 35729 } } peers { permanent_uuid: "09562dcffc8540e39c63a7632649b6d6" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 33565 } }
I20260812 09:45:11.015468 15486 raft_consensus.cc:374] T d214197a770048aebdc4c61a29d9b83e P 09562dcffc8540e39c63a7632649b6d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:45:11.015524 15486 raft_consensus.cc:729] T d214197a770048aebdc4c61a29d9b83e P 09562dcffc8540e39c63a7632649b6d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09562dcffc8540e39c63a7632649b6d6, State: Initialized, Role: FOLLOWER
I20260812 09:45:11.016070 15486 consensus_queue.cc:260] T d214197a770048aebdc4c61a29d9b83e P 09562dcffc8540e39c63a7632649b6d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d9f2d3f220e4cbc9bee179bd669d07f" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 45119 } } peers { permanent_uuid: "f951deec42b44fa88c83cb3aae72b7b6" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 35729 } } peers { permanent_uuid: "09562dcffc8540e39c63a7632649b6d6" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 33565 } }
I20260812 09:45:11.016413 15484 raft_consensus.cc:348] T d214197a770048aebdc4c61a29d9b83e P f951deec42b44fa88c83cb3aae72b7b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d9f2d3f220e4cbc9bee179bd669d07f" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 45119 } } peers { permanent_uuid: "f951deec42b44fa88c83cb3aae72b7b6" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 35729 } } peers { permanent_uuid: "09562dcffc8540e39c63a7632649b6d6" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 33565 } }
I20260812 09:45:11.016714 15484 raft_consensus.cc:374] T d214197a770048aebdc4c61a29d9b83e P f951deec42b44fa88c83cb3aae72b7b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:45:11.016778 15484 raft_consensus.cc:729] T d214197a770048aebdc4c61a29d9b83e P f951deec42b44fa88c83cb3aae72b7b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f951deec42b44fa88c83cb3aae72b7b6, State: Initialized, Role: FOLLOWER
I20260812 09:45:11.016870 15485 raft_consensus.cc:348] T d214197a770048aebdc4c61a29d9b83e P 1d9f2d3f220e4cbc9bee179bd669d07f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d9f2d3f220e4cbc9bee179bd669d07f" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 45119 } } peers { permanent_uuid: "f951deec42b44fa88c83cb3aae72b7b6" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 35729 } } peers { permanent_uuid: "09562dcffc8540e39c63a7632649b6d6" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 33565 } }
I20260812 09:45:11.017102 15485 raft_consensus.cc:374] T d214197a770048aebdc4c61a29d9b83e P 1d9f2d3f220e4cbc9bee179bd669d07f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:45:11.017160 15485 raft_consensus.cc:729] T d214197a770048aebdc4c61a29d9b83e P 1d9f2d3f220e4cbc9bee179bd669d07f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d9f2d3f220e4cbc9bee179bd669d07f, State: Initialized, Role: FOLLOWER
I20260812 09:45:11.017333 15486 ts_tablet_manager.cc:1428] T d214197a770048aebdc4c61a29d9b83e P 09562dcffc8540e39c63a7632649b6d6: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260812 09:45:11.017414 15225 heartbeater.cc:503] Master 127.13.99.62:35481 was elected leader, sending a full tablet report...
I20260812 09:45:11.017627 15485 consensus_queue.cc:260] T d214197a770048aebdc4c61a29d9b83e P 1d9f2d3f220e4cbc9bee179bd669d07f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d9f2d3f220e4cbc9bee179bd669d07f" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 45119 } } peers { permanent_uuid: "f951deec42b44fa88c83cb3aae72b7b6" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 35729 } } peers { permanent_uuid: "09562dcffc8540e39c63a7632649b6d6" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 33565 } }
I20260812 09:45:11.018345 14995 heartbeater.cc:503] Master 127.13.99.62:35481 was elected leader, sending a full tablet report...
I20260812 09:45:11.017222 15484 consensus_queue.cc:260] T d214197a770048aebdc4c61a29d9b83e P f951deec42b44fa88c83cb3aae72b7b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d9f2d3f220e4cbc9bee179bd669d07f" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 45119 } } peers { permanent_uuid: "f951deec42b44fa88c83cb3aae72b7b6" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 35729 } } peers { permanent_uuid: "09562dcffc8540e39c63a7632649b6d6" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 33565 } }
I20260812 09:45:11.019272 15456 heartbeater.cc:503] Master 127.13.99.62:35481 was elected leader, sending a full tablet report...
I20260812 09:45:11.019601 15484 ts_tablet_manager.cc:1428] T d214197a770048aebdc4c61a29d9b83e P f951deec42b44fa88c83cb3aae72b7b6: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
I20260812 09:45:11.021044 15485 ts_tablet_manager.cc:1428] T d214197a770048aebdc4c61a29d9b83e P 1d9f2d3f220e4cbc9bee179bd669d07f: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20260812 09:45:11.053249 15494 raft_consensus.cc:482] T d214197a770048aebdc4c61a29d9b83e P 09562dcffc8540e39c63a7632649b6d6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 09:45:11.053467 15494 raft_consensus.cc:504] T d214197a770048aebdc4c61a29d9b83e P 09562dcffc8540e39c63a7632649b6d6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d9f2d3f220e4cbc9bee179bd669d07f" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 45119 } } peers { permanent_uuid: "f951deec42b44fa88c83cb3aae72b7b6" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 35729 } } peers { permanent_uuid: "09562dcffc8540e39c63a7632649b6d6" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 33565 } }
I20260812 09:45:11.054653 15494 leader_election.cc:288] T d214197a770048aebdc4c61a29d9b83e P 09562dcffc8540e39c63a7632649b6d6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d9f2d3f220e4cbc9bee179bd669d07f (127.13.99.1:45119), f951deec42b44fa88c83cb3aae72b7b6 (127.13.99.3:35729)
I20260812 09:45:11.059536 15479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:11.055021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.2:55017 (local address 127.13.99.1:45119)
0812 09:45:11.055143 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:11.055147 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:11.055506 (+ 359us) server_negotiation.cc:407] Connection header received
0812 09:45:11.055724 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:11.055726 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:11.055768 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:11.055807 (+ 39us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:45:11.056145 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:11.056898 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:11.058343 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:11.058935 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:11.059045 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:11.059107 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:11.059336 (+ 229us) server_negotiation.cc:299] Negotiation successful
0812 09:45:11.059408 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260812 09:45:11.059544 15480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:11.054993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.2:33507 (local address 127.13.99.3:35729)
0812 09:45:11.055137 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:11.055144 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:11.055205 (+ 61us) server_negotiation.cc:407] Connection header received
0812 09:45:11.055377 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:11.055397 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:11.055480 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:11.055565 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:45:11.055856 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:11.056657 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:11.058248 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:11.058882 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:11.059008 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:11.059076 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:11.059336 (+ 260us) server_negotiation.cc:299] Negotiation successful
0812 09:45:11.059408 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 09:45:11.059569 15228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:11.054908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.3:35729 (local address 127.13.99.2:33507)
0812 09:45:11.055125 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:11.055141 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:11.055249 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:11.055576 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:11.055579 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:11.055597 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:45:11.055740 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:11.055747 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:11.056821 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:11.056828 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:11.058120 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:11.058131 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:11.059033 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:11.059037 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:11.059184 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:11.059219 (+ 35us) client_negotiation.cc:769] Sending connection context
0812 09:45:11.059298 (+ 79us) client_negotiation.cc:240] Negotiation successful
0812 09:45:11.059385 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":53}
I20260812 09:45:11.059577 15500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:11.054909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.1:45119 (local address 127.13.99.2:55017)
0812 09:45:11.055385 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:11.055397 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:11.055589 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:11.055916 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:11.055918 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:11.055931 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:45:11.056028 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:11.056032 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:11.057020 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:11.057023 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:11.058219 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:11.058228 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:11.059037 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:11.059040 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:11.059165 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:11.059217 (+ 52us) client_negotiation.cc:769] Sending connection context
0812 09:45:11.059298 (+ 81us) client_negotiation.cc:240] Negotiation successful
0812 09:45:11.059385 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405,"mutex_wait_us":101,"thread_start_us":157,"threads_started":1}
I20260812 09:45:11.060534 14917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d214197a770048aebdc4c61a29d9b83e" candidate_uuid: "09562dcffc8540e39c63a7632649b6d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d9f2d3f220e4cbc9bee179bd669d07f" is_pre_election: true
I20260812 09:45:11.060742 15378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d214197a770048aebdc4c61a29d9b83e" candidate_uuid: "09562dcffc8540e39c63a7632649b6d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f951deec42b44fa88c83cb3aae72b7b6" is_pre_election: true
I20260812 09:45:11.061028 15378 raft_consensus.cc:2454] T d214197a770048aebdc4c61a29d9b83e P f951deec42b44fa88c83cb3aae72b7b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09562dcffc8540e39c63a7632649b6d6 in term 0.
I20260812 09:45:11.061014 14917 raft_consensus.cc:2454] T d214197a770048aebdc4c61a29d9b83e P 1d9f2d3f220e4cbc9bee179bd669d07f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09562dcffc8540e39c63a7632649b6d6 in term 0.
I20260812 09:45:11.061585 15026 leader_election.cc:302] T d214197a770048aebdc4c61a29d9b83e P 09562dcffc8540e39c63a7632649b6d6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09562dcffc8540e39c63a7632649b6d6, 1d9f2d3f220e4cbc9bee179bd669d07f; no voters:
I20260812 09:45:11.061873 15494 raft_consensus.cc:2781] T d214197a770048aebdc4c61a29d9b83e P 09562dcffc8540e39c63a7632649b6d6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 09:45:11.061964 15494 raft_consensus.cc:482] T d214197a770048aebdc4c61a29d9b83e P 09562dcffc8540e39c63a7632649b6d6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 09:45:11.062039 15494 raft_consensus.cc:3037] T d214197a770048aebdc4c61a29d9b83e P 09562dcffc8540e39c63a7632649b6d6 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:45:11.063073 15494 raft_consensus.cc:504] T d214197a770048aebdc4c61a29d9b83e P 09562dcffc8540e39c63a7632649b6d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d9f2d3f220e4cbc9bee179bd669d07f" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 45119 } } peers { permanent_uuid: "f951deec42b44fa88c83cb3aae72b7b6" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 35729 } } peers { permanent_uuid: "09562dcffc8540e39c63a7632649b6d6" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 33565 } }
I20260812 09:45:11.063457 15494 leader_election.cc:288] T d214197a770048aebdc4c61a29d9b83e P 09562dcffc8540e39c63a7632649b6d6 [CANDIDATE]: Term 1 election: Requested vote from peers 1d9f2d3f220e4cbc9bee179bd669d07f (127.13.99.1:45119), f951deec42b44fa88c83cb3aae72b7b6 (127.13.99.3:35729)
I20260812 09:45:11.063745 14917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d214197a770048aebdc4c61a29d9b83e" candidate_uuid: "09562dcffc8540e39c63a7632649b6d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d9f2d3f220e4cbc9bee179bd669d07f"
I20260812 09:45:11.063839 15378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d214197a770048aebdc4c61a29d9b83e" candidate_uuid: "09562dcffc8540e39c63a7632649b6d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f951deec42b44fa88c83cb3aae72b7b6"
I20260812 09:45:11.063859 14917 raft_consensus.cc:3037] T d214197a770048aebdc4c61a29d9b83e P 1d9f2d3f220e4cbc9bee179bd669d07f [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:45:11.063966 15378 raft_consensus.cc:3037] T d214197a770048aebdc4c61a29d9b83e P f951deec42b44fa88c83cb3aae72b7b6 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:45:11.065088 14917 raft_consensus.cc:2454] T d214197a770048aebdc4c61a29d9b83e P 1d9f2d3f220e4cbc9bee179bd669d07f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09562dcffc8540e39c63a7632649b6d6 in term 1.
I20260812 09:45:11.065274 15378 raft_consensus.cc:2454] T d214197a770048aebdc4c61a29d9b83e P f951deec42b44fa88c83cb3aae72b7b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09562dcffc8540e39c63a7632649b6d6 in term 1.
I20260812 09:45:11.065421 15026 leader_election.cc:302] T d214197a770048aebdc4c61a29d9b83e P 09562dcffc8540e39c63a7632649b6d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09562dcffc8540e39c63a7632649b6d6, 1d9f2d3f220e4cbc9bee179bd669d07f; no voters:
I20260812 09:45:11.065618 15494 raft_consensus.cc:2781] T d214197a770048aebdc4c61a29d9b83e P 09562dcffc8540e39c63a7632649b6d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 09:45:11.065858 15494 raft_consensus.cc:686] T d214197a770048aebdc4c61a29d9b83e P 09562dcffc8540e39c63a7632649b6d6 [term 1 LEADER]: Becoming Leader. State: Replica: 09562dcffc8540e39c63a7632649b6d6, State: Running, Role: LEADER
I20260812 09:45:11.066172 15494 consensus_queue.cc:237] T d214197a770048aebdc4c61a29d9b83e P 09562dcffc8540e39c63a7632649b6d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d9f2d3f220e4cbc9bee179bd669d07f" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 45119 } } peers { permanent_uuid: "f951deec42b44fa88c83cb3aae72b7b6" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 35729 } } peers { permanent_uuid: "09562dcffc8540e39c63a7632649b6d6" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 33565 } }
I20260812 09:45:11.069584 14650 catalog_manager.cc:5495] T d214197a770048aebdc4c61a29d9b83e P 09562dcffc8540e39c63a7632649b6d6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09562dcffc8540e39c63a7632649b6d6 (127.13.99.2). New cstate: current_term: 1 leader_uuid: "09562dcffc8540e39c63a7632649b6d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d9f2d3f220e4cbc9bee179bd669d07f" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 45119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f951deec42b44fa88c83cb3aae72b7b6" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 35729 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09562dcffc8540e39c63a7632649b6d6" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 33565 } health_report { overall_health: HEALTHY } } }
Aug 12 09:45:11 dist-test-slave-jq4t krb5kdc[14568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527909, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.99.1@KRBTEST.COM
I20260812 09:45:11.094004 15479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:11.085214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50016 (local address 127.13.99.1:45119)
0812 09:45:11.085372 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:11.085379 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:11.085557 (+ 178us) server_negotiation.cc:407] Connection header received
0812 09:45:11.085724 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:11.085728 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:11.085776 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:11.085864 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:11.086645 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:11.087142 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:11.087982 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:11.088206 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:11.088315 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:11.090887 (+ 2572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:11.090899 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:11.090913 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:11.090936 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:11.092436 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:11.092893 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:11.092898 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:11.092899 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:11.092981 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:11.093215 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:11.093218 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:11.093219 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:11.093446 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:11.093604 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:11.093779 (+ 175us) server_negotiation.cc:299] Negotiation successful
0812 09:45:11.093876 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
Aug 12 09:45:11 dist-test-slave-jq4t krb5kdc[14568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527911, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 09:45:11 dist-test-slave-jq4t krb5kdc[14568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527911, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.13.99.1@KRBTEST.COM
I20260812 09:45:11.125124 15479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:11.113963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50028 (local address 127.13.99.1:45119)
0812 09:45:11.114130 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:11.114135 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:11.114268 (+ 133us) server_negotiation.cc:407] Connection header received
0812 09:45:11.114468 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:11.114471 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:11.114513 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:11.114603 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:11.115350 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:11.116184 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:11.117612 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:11.117839 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:11.117977 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:11.120923 (+ 2946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:11.120930 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:11.120935 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:11.120970 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:11.122902 (+ 1932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:11.123358 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:11.123364 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:11.123368 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:11.123446 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:11.123709 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:11.123714 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:11.123718 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:11.123910 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:11.124028 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:11.124780 (+ 752us) server_negotiation.cc:299] Negotiation successful
0812 09:45:11.124903 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
Aug 12 09:45:11 dist-test-slave-jq4t krb5kdc[14568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527911, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 09:45:11 dist-test-slave-jq4t krb5kdc[14568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527911, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.99.1@KRBTEST.COM
I20260812 09:45:11.153779 15479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:11.145456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50036 (local address 127.13.99.1:45119)
0812 09:45:11.145596 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:11.145601 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:11.145758 (+ 157us) server_negotiation.cc:407] Connection header received
0812 09:45:11.145921 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:11.145929 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:11.145994 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:11.146085 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:11.146771 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:11.147270 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:11.148109 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:11.148325 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:11.148433 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:11.150820 (+ 2387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:11.150823 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:11.150825 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:11.150849 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:11.152245 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:11.152759 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:11.152762 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:11.152763 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:11.152836 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:11.153104 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:11.153106 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:11.153108 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:11.153265 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:11.153362 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:11.153556 (+ 194us) server_negotiation.cc:299] Negotiation successful
0812 09:45:11.153653 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260812 09:45:11.155624 14851 tablet_service.cc:2908] Rejecting scan request for tablet d214197a770048aebdc4c61a29d9b83e: Uninitialized: clean time has not yet been initialized
I20260812 09:45:11.158244 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 14769
I20260812 09:45:11.177770 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 15001
W20260812 09:45:11.186450 15457 tablet.cc:2369] T d214197a770048aebdc4c61a29d9b83e P f951deec42b44fa88c83cb3aae72b7b6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 09:45:11.187960 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 15231
I20260812 09:45:11.197132 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 14595
2026-08-12T09:45:11Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2571 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 09:45:11 dist-test-slave-jq4t krb5kdc[15521](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 09:45:11 dist-test-slave-jq4t krb5kdc[15521](info): set up 2 sockets
Aug 12 09:45:11 dist-test-slave-jq4t krb5kdc[15521](info): commencing operation
krb5kdc: starting...
W20260812 09:45:12.476462 13708 mini_kdc.cc:121] Time spent starting KDC: real 1.252s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/krb5kdc/test-user.keytab.
Aug 12 09:45:12 dist-test-slave-jq4t krb5kdc[15521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527912, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T09:45:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T09:45:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.99.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.99.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.99.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab.
Entry for principal kudu/127.13.99.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab.
I20260812 09:45:12.601735 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.62:32955
--webserver_interface=127.13.99.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab
--principal=kudu/127.13.99.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:45657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.99.62:32955
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:45:12.715948 15544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:45:12.716218 15544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:45:12.716259 15544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:45:12.720285 15544 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 09:45:12.720350 15544 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260812 09:45:12.720367 15544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:45:12.720386 15544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:45:12.720402 15544 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 09:45:12.720461 15544 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 09:45:12.725826 15544 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:45657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.99.62:32955
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab
--principal=kudu/127.13.99.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.62:32955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.99.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15544
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:45:12.726871 15544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:45:12.727936 15544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:45:12.736411 15553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:12.736403 15552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:12.736402 15555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:45:12.736771 15544 server_base.cc:1034] running on GCE node
I20260812 09:45:12.737357 15544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:45:12.738445 15544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:45:12.739605 15544 hybrid_clock.cc:648] HybridClock initialized: now 1786527912739586 us; error 29 us; skew 500 ppm
Aug 12 09:45:12 dist-test-slave-jq4t krb5kdc[15521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527912, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:45:12.742494 15544 init.cc:377] Logged in from keytab as kudu/127.13.99.62@KRBTEST.COM (short username kudu)
I20260812 09:45:12.743399 15544 webserver.cc:460] Webserver started at http://127.13.99.62:42355/ using document root <none> and password file <none>
I20260812 09:45:12.743957 15544 fs_manager.cc:359] Metadata directory not provided
I20260812 09:45:12.744020 15544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:45:12.744266 15544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:45:12.746337 15544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/minicluster-data/master-0/data/instance:
uuid: "3eac478794a6474684ce396fa014045f"
format_stamp: "Formatted at 2026-08-12 09:45:12 on dist-test-slave-jq4t"
server_key: "16e43bccc1576a59235249fc64b0345c"
server_key_iv: "a4526e17ba8411d0c88a240966a0af83"
server_key_version: "encryptionkey@0"
I20260812 09:45:12.746953 15544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/minicluster-data/master-0/wal/instance:
uuid: "3eac478794a6474684ce396fa014045f"
format_stamp: "Formatted at 2026-08-12 09:45:12 on dist-test-slave-jq4t"
server_key: "16e43bccc1576a59235249fc64b0345c"
server_key_iv: "a4526e17ba8411d0c88a240966a0af83"
server_key_version: "encryptionkey@0"
I20260812 09:45:12.750558 15544 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260812 09:45:12.752983 15564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:12.754204 15544 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 09:45:12.754305 15544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/minicluster-data/master-0/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/minicluster-data/master-0/wal
uuid: "3eac478794a6474684ce396fa014045f"
format_stamp: "Formatted at 2026-08-12 09:45:12 on dist-test-slave-jq4t"
server_key: "16e43bccc1576a59235249fc64b0345c"
server_key_iv: "a4526e17ba8411d0c88a240966a0af83"
server_key_version: "encryptionkey@0"
I20260812 09:45:12.754438 15544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:45:12.772630 15544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:45:12.776101 15544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:45:12.776316 15544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:45:12.784963 15544 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.62:32955
I20260812 09:45:12.784987 15667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.62:32955 every 8 connection(s)
I20260812 09:45:12.785943 15544 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
I20260812 09:45:12.788061 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 15544
I20260812 09:45:12.788185 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/minicluster-data/master-0/wal/instance
I20260812 09:45:12.788466 13708 external_mini_cluster.cc:1428] Setting key 3cce11e6eb7d4073097863d64e9a1e76
I20260812 09:45:12.789171 15668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:45:12.795095 15668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3eac478794a6474684ce396fa014045f: Bootstrap starting.
Aug 12 09:45:12 dist-test-slave-jq4t krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527912, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
I20260812 09:45:12.797441 15668 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3eac478794a6474684ce396fa014045f: Neither blocks nor log segments found. Creating new log.
I20260812 09:45:12.798221 15668 log.cc:824] T 00000000000000000000000000000000 P 3eac478794a6474684ce396fa014045f: Log is configured to *not* fsync() on all Append() calls
I20260812 09:45:12.800263 15668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3eac478794a6474684ce396fa014045f: No bootstrap required, opened a new log
I20260812 09:45:12.803400 15671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:12.789968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46580 (local address 127.13.99.62:32955)
0812 09:45:12.790406 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:12.790421 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:12.790458 (+ 37us) server_negotiation.cc:407] Connection header received
0812 09:45:12.791127 (+ 669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:12.791144 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:12.791461 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:12.791809 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:12.792695 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:12.793687 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:12.794557 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:12.795032 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:12.795207 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:12.797866 (+ 2659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:12.797871 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:12.797885 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:12.797913 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:12.800610 (+ 2697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:12.801062 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:12.801068 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:12.801074 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:12.801196 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:12.801493 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:12.801497 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:12.801498 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:12.801898 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:12.802109 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:12.802405 (+ 296us) server_negotiation.cc:299] Negotiation successful
0812 09:45:12.802699 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":138,"threads_started":1}
I20260812 09:45:12.803295 15668 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3eac478794a6474684ce396fa014045f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3eac478794a6474684ce396fa014045f" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 32955 } }
I20260812 09:45:12.803530 15668 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3eac478794a6474684ce396fa014045f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:45:12.803580 15668 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3eac478794a6474684ce396fa014045f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3eac478794a6474684ce396fa014045f, State: Initialized, Role: FOLLOWER
I20260812 09:45:12.803992 15668 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3eac478794a6474684ce396fa014045f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3eac478794a6474684ce396fa014045f" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 32955 } }
I20260812 09:45:12.804101 15668 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3eac478794a6474684ce396fa014045f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 09:45:12.804140 15668 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3eac478794a6474684ce396fa014045f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 09:45:12.804201 15668 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3eac478794a6474684ce396fa014045f [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:45:12.805089 15668 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3eac478794a6474684ce396fa014045f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3eac478794a6474684ce396fa014045f" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 32955 } }
I20260812 09:45:12.805435 15668 leader_election.cc:302] T 00000000000000000000000000000000 P 3eac478794a6474684ce396fa014045f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3eac478794a6474684ce396fa014045f; no voters:
I20260812 09:45:12.805835 15668 leader_election.cc:288] T 00000000000000000000000000000000 P 3eac478794a6474684ce396fa014045f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 09:45:12.805891 15675 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3eac478794a6474684ce396fa014045f [term 1 FOLLOWER]: Leader election won for term 1
I20260812 09:45:12.806138 15675 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3eac478794a6474684ce396fa014045f [term 1 LEADER]: Becoming Leader. State: Replica: 3eac478794a6474684ce396fa014045f, State: Running, Role: LEADER
I20260812 09:45:12.806440 15675 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3eac478794a6474684ce396fa014045f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3eac478794a6474684ce396fa014045f" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 32955 } }
I20260812 09:45:12.807147 15668 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3eac478794a6474684ce396fa014045f [sys.catalog]: configured and running, proceeding with master startup.
I20260812 09:45:12.808408 15678 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3eac478794a6474684ce396fa014045f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3eac478794a6474684ce396fa014045f. Latest consensus state: current_term: 1 leader_uuid: "3eac478794a6474684ce396fa014045f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3eac478794a6474684ce396fa014045f" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 32955 } } }
I20260812 09:45:12.808576 15678 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3eac478794a6474684ce396fa014045f [sys.catalog]: This master's current role is: LEADER
I20260812 09:45:12.808393 15676 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3eac478794a6474684ce396fa014045f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3eac478794a6474684ce396fa014045f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3eac478794a6474684ce396fa014045f" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 32955 } } }
I20260812 09:45:12.808799 15676 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3eac478794a6474684ce396fa014045f [sys.catalog]: This master's current role is: LEADER
W20260812 09:45:12.811939 15700 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3eac478794a6474684ce396fa014045f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 09:45:12.812008 15700 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 09:45:12.812101 15686 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 09:45:12.812994 15686 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 09:45:12.818634 15686 catalog_manager.cc:1347] Generated new cluster ID: 5232b2249b804962a1b79cdd6297f5c8
I20260812 09:45:12.818710 15686 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 09:45:12.834307 15686 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 09:45:12.835443 15686 catalog_manager.cc:1504] Loading token signing keys...
I20260812 09:45:12.843631 15686 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3eac478794a6474684ce396fa014045f: Generated new TSK 0
I20260812 09:45:12.844352 15686 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.99.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.99.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.99.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.1.keytab.
Entry for principal kudu/127.13.99.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.1.keytab.
I20260812 09:45:12.879218 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.99.1:0
--local_ip_for_outbound_sockets=127.13.99.1
--webserver_interface=127.13.99.1
--webserver_port=0
--tserver_master_addrs=127.13.99.62:32955
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.1.keytab
--principal=kudu/127.13.99.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:45657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:45:12.992856 15706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:45:12.993149 15706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:45:12.993224 15706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:45:12.997345 15706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:45:12.997427 15706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:45:12.997494 15706 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 09:45:12.997584 15706 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.99.1
I20260812 09:45:13.002815 15706 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:45657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.1.keytab
--principal=kudu/127.13.99.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.99.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.99.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.99.62:32955
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15706
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.99.1
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:45:13.003947 15706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:45:13.005090 15706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:45:13.015121 15720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:13.015131 15714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:45:13.015447 15706 server_base.cc:1034] running on GCE node
W20260812 09:45:13.015226 15715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:45:13.015898 15706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:45:13.016595 15706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:45:13.017781 15706 hybrid_clock.cc:648] HybridClock initialized: now 1786527913017768 us; error 30 us; skew 500 ppm
Aug 12 09:45:13 dist-test-slave-jq4t krb5kdc[15521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527913, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:45:13.020548 15706 init.cc:377] Logged in from keytab as kudu/127.13.99.1@KRBTEST.COM (short username kudu)
I20260812 09:45:13.021384 15706 webserver.cc:460] Webserver started at http://127.13.99.1:43209/ using document root <none> and password file <none>
I20260812 09:45:13.021948 15706 fs_manager.cc:359] Metadata directory not provided
I20260812 09:45:13.022042 15706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:45:13.022282 15706 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:45:13.024427 15706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/minicluster-data/ts-0/data/instance:
uuid: "981793cd97244da8b9e2b369c1892d32"
format_stamp: "Formatted at 2026-08-12 09:45:13 on dist-test-slave-jq4t"
server_key: "924973851f1b5da0d658629d9bd3d673"
server_key_iv: "c27825829d9dbc99dec87051f4557463"
server_key_version: "encryptionkey@0"
I20260812 09:45:13.025061 15706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/minicluster-data/ts-0/wal/instance:
uuid: "981793cd97244da8b9e2b369c1892d32"
format_stamp: "Formatted at 2026-08-12 09:45:13 on dist-test-slave-jq4t"
server_key: "924973851f1b5da0d658629d9bd3d673"
server_key_iv: "c27825829d9dbc99dec87051f4557463"
server_key_version: "encryptionkey@0"
I20260812 09:45:13.028905 15706 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 09:45:13.031284 15732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:13.032552 15706 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 09:45:13.032680 15706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/minicluster-data/ts-0/wal
uuid: "981793cd97244da8b9e2b369c1892d32"
format_stamp: "Formatted at 2026-08-12 09:45:13 on dist-test-slave-jq4t"
server_key: "924973851f1b5da0d658629d9bd3d673"
server_key_iv: "c27825829d9dbc99dec87051f4557463"
server_key_version: "encryptionkey@0"
I20260812 09:45:13.032785 15706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:45:13.052637 15706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:45:13.055578 15706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:45:13.055790 15706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:45:13.057471 15706 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:45:13.057540 15706 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:13.057608 15706 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:45:13.057660 15706 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 09:45:13 dist-test-slave-jq4t krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527913, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.1@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
I20260812 09:45:13.069433 15706 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.1:33687
I20260812 09:45:13.069535 15936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.1:33687 every 8 connection(s)
I20260812 09:45:13.070379 15706 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/minicluster-data/ts-0/data/info.pb
I20260812 09:45:13.071949 15671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:13.058316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.1:52811 (local address 127.13.99.62:32955)
0812 09:45:13.058475 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:13.058480 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:13.059549 (+ 1069us) server_negotiation.cc:407] Connection header received
0812 09:45:13.060791 (+ 1242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:13.060798 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:13.060874 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:13.060993 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:13.062506 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:13.063060 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:13.064546 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:13.064801 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:13.064922 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:13.067874 (+ 2952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:13.067878 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:13.067880 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:13.067911 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:13.069714 (+ 1803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:13.070249 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:13.070253 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:13.070254 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:13.070354 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:13.070746 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:13.070750 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:13.070752 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:13.071020 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:13.071132 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:13.071697 (+ 565us) server_negotiation.cc:299] Negotiation successful
0812 09:45:13.071824 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260812 09:45:13.072782 15750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:13.058882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:32955 (local address 127.13.99.1:52811)
0812 09:45:13.059391 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:13.059434 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:13.060431 (+ 997us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:13.061203 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:13.061213 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:13.061717 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:45:13.062330 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:13.062354 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:13.063196 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:13.063199 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:13.064395 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:13.064405 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:13.065152 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:13.065156 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:13.065301 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:13.065920 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:45:13.065946 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:45:13.067698 (+ 1752us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:45:13.069842 (+ 2144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:13.069848 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:13.069863 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:13.070136 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:13.070453 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:13.070456 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:13.070458 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:13.070640 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:13.071191 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:45:13.071197 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:45:13.071455 (+ 258us) client_negotiation.cc:769] Sending connection context
0812 09:45:13.071620 (+ 165us) client_negotiation.cc:240] Negotiation successful
0812 09:45:13.071874 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":143,"threads_started":1}
I20260812 09:45:13.074150 15940 heartbeater.cc:347] Connected to a master server at 127.13.99.62:32955
I20260812 09:45:13.074324 15940 heartbeater.cc:464] Registering TS with master...
I20260812 09:45:13.074849 15940 heartbeater.cc:511] Master 127.13.99.62:32955 requested a full tablet report, sending...
I20260812 09:45:13.076159 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 15706
I20260812 09:45:13.076277 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/minicluster-data/ts-0/wal/instance
I20260812 09:45:13.076592 13708 external_mini_cluster.cc:1428] Setting key b86359af3531778afc7248b7b1f9fc59
I20260812 09:45:13.076566 15588 ts_manager.cc:194] Registered new tserver with Master: 981793cd97244da8b9e2b369c1892d32 (127.13.99.1:33687)
I20260812 09:45:13.078122 15588 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.99.1@KRBTEST.COM'} at 127.13.99.1:52811
I20260812 09:45:13.086017 15671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:13.076306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.1:34997 (local address 127.13.99.62:32955)
0812 09:45:13.076418 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:13.076422 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:13.077188 (+ 766us) server_negotiation.cc:407] Connection header received
0812 09:45:13.077334 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:13.077337 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:13.077379 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:13.077460 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:13.077936 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:13.078532 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:13.081233 (+ 2701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:13.081460 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:13.081548 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:13.082848 (+ 1300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:13.082851 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:13.082853 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:13.082874 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:13.084318 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:13.084878 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:13.084881 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:13.084882 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:13.084957 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:13.085279 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:13.085281 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:13.085282 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:13.085443 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:13.085516 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:13.085811 (+ 295us) server_negotiation.cc:299] Negotiation successful
0812 09:45:13.085907 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260812 09:45:13.086170 15953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:13.076249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:32955 (local address 127.13.99.1:34997)
0812 09:45:13.077094 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:13.077109 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:13.077211 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:13.077498 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:13.077501 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:13.077707 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:45:13.077817 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:13.077823 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:13.080112 (+ 2289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:13.080118 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:13.081104 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:13.081118 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:13.081583 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:13.081589 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:13.081714 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:13.082212 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:45:13.082231 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:45:13.082707 (+ 476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:45:13.084438 (+ 1731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:13.084444 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:13.084449 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:13.084772 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:13.085060 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:13.085065 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:13.085069 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:13.085181 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:13.085545 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:45:13.085548 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:45:13.085654 (+ 106us) client_negotiation.cc:769] Sending connection context
0812 09:45:13.085760 (+ 106us) client_negotiation.cc:240] Negotiation successful
0812 09:45:13.085925 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":759,"thread_start_us":100,"threads_started":1}
WARNING: no policy specified for kudu/127.13.99.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.99.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.99.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.2.keytab.
Entry for principal kudu/127.13.99.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.2.keytab.
I20260812 09:45:13.107781 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.99.2:0
--local_ip_for_outbound_sockets=127.13.99.2
--webserver_interface=127.13.99.2
--webserver_port=0
--tserver_master_addrs=127.13.99.62:32955
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.2.keytab
--principal=kudu/127.13.99.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:45657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:45:13.221239 15959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:45:13.221513 15959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:45:13.221557 15959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:45:13.225704 15959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:45:13.225755 15959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:45:13.225793 15959 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 09:45:13.225824 15959 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.99.2
I20260812 09:45:13.230777 15959 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:45657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.2.keytab
--principal=kudu/127.13.99.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.99.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.99.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.99.62:32955
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15959
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.99.2
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:45:13.231734 15959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:45:13.232697 15959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:45:13.242702 15973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:45:13.242756 15959 server_base.cc:1034] running on GCE node
W20260812 09:45:13.242699 15975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:13.242915 15972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:45:13.243386 15959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:45:13.244024 15959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:45:13.245219 15959 hybrid_clock.cc:648] HybridClock initialized: now 1786527913245187 us; error 44 us; skew 500 ppm
Aug 12 09:45:13 dist-test-slave-jq4t krb5kdc[15521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527913, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:45:13.248080 15959 init.cc:377] Logged in from keytab as kudu/127.13.99.2@KRBTEST.COM (short username kudu)
I20260812 09:45:13.248965 15959 webserver.cc:460] Webserver started at http://127.13.99.2:44781/ using document root <none> and password file <none>
I20260812 09:45:13.249473 15959 fs_manager.cc:359] Metadata directory not provided
I20260812 09:45:13.249539 15959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:45:13.249701 15959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:45:13.251626 15959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/minicluster-data/ts-1/data/instance:
uuid: "0631d48e67d34bcb84ab4ecb65163865"
format_stamp: "Formatted at 2026-08-12 09:45:13 on dist-test-slave-jq4t"
server_key: "b1f5ee3046fe33b18817b739359a974f"
server_key_iv: "a8d3e5a13933c27c7882b224c5d74b6f"
server_key_version: "encryptionkey@0"
I20260812 09:45:13.252128 15959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/minicluster-data/ts-1/wal/instance:
uuid: "0631d48e67d34bcb84ab4ecb65163865"
format_stamp: "Formatted at 2026-08-12 09:45:13 on dist-test-slave-jq4t"
server_key: "b1f5ee3046fe33b18817b739359a974f"
server_key_iv: "a8d3e5a13933c27c7882b224c5d74b6f"
server_key_version: "encryptionkey@0"
I20260812 09:45:13.256176 15959 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 09:45:13.258730 15992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:13.260099 15959 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 09:45:13.260206 15959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/minicluster-data/ts-1/wal
uuid: "0631d48e67d34bcb84ab4ecb65163865"
format_stamp: "Formatted at 2026-08-12 09:45:13 on dist-test-slave-jq4t"
server_key: "b1f5ee3046fe33b18817b739359a974f"
server_key_iv: "a8d3e5a13933c27c7882b224c5d74b6f"
server_key_version: "encryptionkey@0"
I20260812 09:45:13.260288 15959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:45:13.274681 15959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:45:13.277601 15959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:45:13.277796 15959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:45:13.279286 15959 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:45:13.279364 15959 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:13.279434 15959 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:45:13.279490 15959 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:13.289485 15959 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.2:35803
I20260812 09:45:13.289567 16178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.2:35803 every 8 connection(s)
I20260812 09:45:13.290541 15959 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/minicluster-data/ts-1/data/info.pb
Aug 12 09:45:13 dist-test-slave-jq4t krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527913, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.2@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
I20260812 09:45:13.294727 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 15959
I20260812 09:45:13.294868 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/minicluster-data/ts-1/wal/instance
I20260812 09:45:13.295183 15671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:13.279975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.2:33969 (local address 127.13.99.62:32955)
0812 09:45:13.280135 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:13.280142 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:13.282380 (+ 2238us) server_negotiation.cc:407] Connection header received
0812 09:45:13.283609 (+ 1229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:13.283613 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:13.283687 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:13.283809 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:13.285686 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:13.286441 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:13.288311 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:13.288665 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:13.288752 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:13.291768 (+ 3016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:13.291772 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:13.291774 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:13.291799 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:13.293097 (+ 1298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:13.293552 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:13.293555 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:13.293556 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:13.293631 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:13.294019 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:13.294022 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:13.294024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:13.294212 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:13.294302 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:13.294924 (+ 622us) server_negotiation.cc:299] Negotiation successful
0812 09:45:13.295062 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260812 09:45:13.295182 13708 external_mini_cluster.cc:1428] Setting key 9bdfc41a6cd4199ba23d9d131fb0bd65
I20260812 09:45:13.296141 16005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:13.280984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:32955 (local address 127.13.99.2:33969)
0812 09:45:13.282210 (+ 1226us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:13.282271 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:13.283340 (+ 1069us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:13.284154 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:13.284166 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:13.284757 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:45:13.285501 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:13.285524 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:13.286573 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:13.286579 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:13.288168 (+ 1589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:13.288181 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:13.288818 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:13.288825 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:13.289029 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:13.289677 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:45:13.289697 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:45:13.291598 (+ 1901us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:45:13.293202 (+ 1604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:13.293207 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:13.293217 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:13.293463 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:13.293713 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:13.293715 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:13.293717 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:13.293843 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:13.294312 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:45:13.294317 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:45:13.294595 (+ 278us) client_negotiation.cc:769] Sending connection context
0812 09:45:13.294819 (+ 224us) client_negotiation.cc:240] Negotiation successful
0812 09:45:13.295167 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1029,"thread_start_us":146,"threads_started":1}
I20260812 09:45:13.297441 16179 heartbeater.cc:347] Connected to a master server at 127.13.99.62:32955
I20260812 09:45:13.297601 16179 heartbeater.cc:464] Registering TS with master...
I20260812 09:45:13.298183 16179 heartbeater.cc:511] Master 127.13.99.62:32955 requested a full tablet report, sending...
I20260812 09:45:13.299351 15589 ts_manager.cc:194] Registered new tserver with Master: 0631d48e67d34bcb84ab4ecb65163865 (127.13.99.2:35803)
I20260812 09:45:13.299962 15589 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.99.2@KRBTEST.COM'} at 127.13.99.2:33969
WARNING: no policy specified for kudu/127.13.99.3@KRBTEST.COM; defaulting to no policy
I20260812 09:45:13.308805 16190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:13.300084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:32955 (local address 127.13.99.2:51177)
0812 09:45:13.300519 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:13.300536 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:13.300677 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:13.300998 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:13.301001 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:13.301239 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:45:13.301353 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:13.301361 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:13.302449 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:13.302452 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:13.303204 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:13.303214 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:13.303861 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:13.303864 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:13.303934 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:13.304535 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:45:13.304552 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:45:13.304889 (+ 337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:45:13.306839 (+ 1950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:13.306842 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:13.306843 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:13.307123 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:13.307470 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:13.307472 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:13.307473 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:13.307559 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:13.308102 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:45:13.308105 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:45:13.308235 (+ 130us) client_negotiation.cc:769] Sending connection context
0812 09:45:13.308391 (+ 156us) client_negotiation.cc:240] Negotiation successful
0812 09:45:13.308604 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":239,"threads_started":1}
I20260812 09:45:13.308832 15671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:13.300165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.2:51177 (local address 127.13.99.62:32955)
0812 09:45:13.300294 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:13.300301 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:13.300606 (+ 305us) server_negotiation.cc:407] Connection header received
0812 09:45:13.300801 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:13.300805 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:13.300863 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:13.300953 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:13.301493 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:13.302307 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:13.303325 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:13.303735 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:13.303864 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:13.305031 (+ 1167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:13.305036 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:13.305041 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:13.305074 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:13.306714 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:13.307260 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:13.307266 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:13.307271 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:13.307367 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:13.307660 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:13.307666 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:13.307671 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:13.307999 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:13.308112 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:13.308468 (+ 356us) server_negotiation.cc:299] Negotiation successful
0812 09:45:13.308622 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.99.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.99.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.3.keytab.
Entry for principal kudu/127.13.99.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.3.keytab.
I20260812 09:45:13.328558 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.99.3:0
--local_ip_for_outbound_sockets=127.13.99.3
--webserver_interface=127.13.99.3
--webserver_port=0
--tserver_master_addrs=127.13.99.62:32955
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.3.keytab
--principal=kudu/127.13.99.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:45657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:45:13.446628 16192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:45:13.446902 16192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:45:13.446949 16192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:45:13.450920 16192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:45:13.450973 16192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:45:13.451011 16192 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 09:45:13.451045 16192 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.99.3
I20260812 09:45:13.456213 16192 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:45657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.3.keytab
--principal=kudu/127.13.99.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.99.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.99.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.99.62:32955
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16192
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.99.3
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:45:13.457310 16192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:45:13.458386 16192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:45:13.468192 16204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:13.468192 16200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:13.468184 16202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:45:13.468392 16192 server_base.cc:1034] running on GCE node
I20260812 09:45:13.468909 16192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:45:13.469556 16192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:45:13.470763 16192 hybrid_clock.cc:648] HybridClock initialized: now 1786527913470726 us; error 50 us; skew 500 ppm
Aug 12 09:45:13 dist-test-slave-jq4t krb5kdc[15521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527913, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:45:13.473703 16192 init.cc:377] Logged in from keytab as kudu/127.13.99.3@KRBTEST.COM (short username kudu)
I20260812 09:45:13.474663 16192 webserver.cc:460] Webserver started at http://127.13.99.3:35791/ using document root <none> and password file <none>
I20260812 09:45:13.475291 16192 fs_manager.cc:359] Metadata directory not provided
I20260812 09:45:13.475396 16192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:45:13.475632 16192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:45:13.477802 16192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/minicluster-data/ts-2/data/instance:
uuid: "54f7fe0d3da94da18132fb2d7b0ed0b8"
format_stamp: "Formatted at 2026-08-12 09:45:13 on dist-test-slave-jq4t"
server_key: "30bf941f290a74592340e66846a68732"
server_key_iv: "76ac90464c3e0d7feab6efe8adbadbac"
server_key_version: "encryptionkey@0"
I20260812 09:45:13.478399 16192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/minicluster-data/ts-2/wal/instance:
uuid: "54f7fe0d3da94da18132fb2d7b0ed0b8"
format_stamp: "Formatted at 2026-08-12 09:45:13 on dist-test-slave-jq4t"
server_key: "30bf941f290a74592340e66846a68732"
server_key_iv: "76ac90464c3e0d7feab6efe8adbadbac"
server_key_version: "encryptionkey@0"
I20260812 09:45:13.482096 16192 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260812 09:45:13.484452 16217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:13.485688 16192 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 09:45:13.485826 16192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/minicluster-data/ts-2/wal
uuid: "54f7fe0d3da94da18132fb2d7b0ed0b8"
format_stamp: "Formatted at 2026-08-12 09:45:13 on dist-test-slave-jq4t"
server_key: "30bf941f290a74592340e66846a68732"
server_key_iv: "76ac90464c3e0d7feab6efe8adbadbac"
server_key_version: "encryptionkey@0"
I20260812 09:45:13.485937 16192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:45:13.504215 16192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:45:13.507329 16192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:45:13.507545 16192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:45:13.509323 16192 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:45:13.509426 16192 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:13.509502 16192 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:45:13.509536 16192 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 09:45:13 dist-test-slave-jq4t krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527913, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.3@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
I20260812 09:45:13.520892 16192 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.3:38427
I20260812 09:45:13.520985 16417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.3:38427 every 8 connection(s)
I20260812 09:45:13.521867 16192 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/minicluster-data/ts-2/data/info.pb
I20260812 09:45:13.523154 15671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:13.509995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.3:41633 (local address 127.13.99.62:32955)
0812 09:45:13.510167 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:13.510173 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:13.511537 (+ 1364us) server_negotiation.cc:407] Connection header received
0812 09:45:13.512640 (+ 1103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:13.512644 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:13.512707 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:13.512786 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:13.514003 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:13.514557 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:13.515792 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:13.516102 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:13.516224 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:13.519206 (+ 2982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:13.519210 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:13.519213 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:13.519249 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:13.520894 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:13.521502 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:13.521509 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:13.521512 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:13.521611 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:13.522007 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:13.522010 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:13.522012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:13.522208 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:13.522321 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:13.522920 (+ 599us) server_negotiation.cc:299] Negotiation successful
0812 09:45:13.523031 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260812 09:45:13.524044 16235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:13.510888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:32955 (local address 127.13.99.3:41633)
0812 09:45:13.511363 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:13.511415 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:13.512376 (+ 961us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:13.512971 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:13.512980 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:13.513365 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:45:13.513850 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:13.513863 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:13.514694 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:13.514698 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:13.515645 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:13.515659 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:13.516278 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:13.516282 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:13.516530 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:13.517275 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:45:13.517299 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:45:13.519021 (+ 1722us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:45:13.521082 (+ 2061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:13.521087 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:13.521103 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:13.521383 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:13.521716 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:13.521719 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:13.521721 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:13.521871 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:13.522320 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:45:13.522328 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:45:13.522637 (+ 309us) client_negotiation.cc:769] Sending connection context
0812 09:45:13.522850 (+ 213us) client_negotiation.cc:240] Negotiation successful
0812 09:45:13.523123 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":139,"threads_started":1}
I20260812 09:45:13.525296 16419 heartbeater.cc:347] Connected to a master server at 127.13.99.62:32955
I20260812 09:45:13.525518 16419 heartbeater.cc:464] Registering TS with master...
I20260812 09:45:13.526015 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 16192
I20260812 09:45:13.526068 16419 heartbeater.cc:511] Master 127.13.99.62:32955 requested a full tablet report, sending...
I20260812 09:45:13.526163 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/minicluster-data/ts-2/wal/instance
I20260812 09:45:13.526531 13708 external_mini_cluster.cc:1428] Setting key 1a95be3503205e73096acc426c8cad18
I20260812 09:45:13.527288 15588 ts_manager.cc:194] Registered new tserver with Master: 54f7fe0d3da94da18132fb2d7b0ed0b8 (127.13.99.3:38427)
I20260812 09:45:13.527992 15588 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.99.3@KRBTEST.COM'} at 127.13.99.3:41633
I20260812 09:45:13.528718 13708 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 09:45:13.536067 15671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:13.528031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.3:55349 (local address 127.13.99.62:32955)
0812 09:45:13.528158 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:13.528165 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:13.528845 (+ 680us) server_negotiation.cc:407] Connection header received
0812 09:45:13.529224 (+ 379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:13.529228 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:13.529280 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:13.529329 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:13.529880 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:13.530406 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:13.531260 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:13.531491 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:13.531593 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:13.532809 (+ 1216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:13.532812 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:13.532815 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:13.532840 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:13.534387 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:13.534821 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:13.534825 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:13.534827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:13.534916 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:13.535202 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:13.535206 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:13.535208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:13.535413 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:13.535524 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:13.535787 (+ 263us) server_negotiation.cc:299] Negotiation successful
0812 09:45:13.535911 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260812 09:45:13.536078 16431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:13.527984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:32955 (local address 127.13.99.3:55349)
0812 09:45:13.528744 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:13.528757 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:13.529068 (+ 311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:13.529471 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:13.529473 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:13.529657 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:45:13.529751 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:13.529756 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:13.530536 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:13.530539 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:13.531126 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:13.531134 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:13.531608 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:13.531612 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:13.531712 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:13.532260 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:45:13.532272 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:45:13.532666 (+ 394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:45:13.534499 (+ 1833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:13.534503 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:13.534505 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:13.534710 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:13.535011 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:13.535015 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:13.535017 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:13.535108 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:13.535513 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:45:13.535516 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:45:13.535619 (+ 103us) client_negotiation.cc:769] Sending connection context
0812 09:45:13.535731 (+ 112us) client_negotiation.cc:240] Negotiation successful
0812 09:45:13.535886 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":690,"thread_start_us":200,"threads_started":1}
Aug 12 09:45:13 dist-test-slave-jq4t krb5kdc[15521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527913, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 09:45:13 dist-test-slave-jq4t krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527913, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
I20260812 09:45:13.556588 15671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:13.548721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46588 (local address 127.13.99.62:32955)
0812 09:45:13.548865 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:13.548868 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:13.548883 (+ 15us) server_negotiation.cc:407] Connection header received
0812 09:45:13.548923 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:13.548925 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:13.548962 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:13.549046 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:13.549715 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:13.550228 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:13.551062 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:13.551266 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:13.551341 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:13.553756 (+ 2415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:13.553759 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:13.553761 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:13.553785 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:13.555086 (+ 1301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:13.555446 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:13.555449 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:13.555450 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:13.555524 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:13.555751 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:13.555753 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:13.555754 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:13.555915 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:13.556016 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:13.556247 (+ 231us) server_negotiation.cc:299] Negotiation successful
0812 09:45:13.556339 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 09:45:13.559608 15589 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46588:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 09:45:13.562775 15589 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 09:45:13.577286 16454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:13.571260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.1:33687 (local address 127.0.0.1:51680)
0812 09:45:13.572098 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:13.572148 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:13.572273 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:13.573068 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:13.573074 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:13.573097 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:45:13.573230 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:13.573236 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:13.574320 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:13.574326 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:13.575592 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:13.575604 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:13.576732 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:13.576736 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:13.576844 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:13.576882 (+ 38us) client_negotiation.cc:769] Sending connection context
0812 09:45:13.576998 (+ 116us) client_negotiation.cc:240] Negotiation successful
0812 09:45:13.577128 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":649,"thread_start_us":83,"threads_started":1}
I20260812 09:45:13.577332 16449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:13.571617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.2:35803 (local address 127.0.0.1:59222)
0812 09:45:13.572508 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:13.572521 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:13.572606 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:13.573238 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:13.573241 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:13.573258 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:45:13.573369 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:13.573375 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:13.574945 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:13.574948 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:13.576009 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:13.576018 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:13.577010 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:13.577013 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:13.577093 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:13.577115 (+ 22us) client_negotiation.cc:769] Sending connection context
0812 09:45:13.577175 (+ 60us) client_negotiation.cc:240] Negotiation successful
0812 09:45:13.577229 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":811,"thread_start_us":99,"threads_started":1}
I20260812 09:45:13.577658 16455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:13.572046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59222 (local address 127.13.99.2:35803)
0812 09:45:13.572600 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:13.572614 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:13.572638 (+ 24us) server_negotiation.cc:407] Connection header received
0812 09:45:13.572719 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:13.572725 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:13.572926 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:13.573085 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:45:13.573463 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:13.574701 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:13.576127 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:13.576895 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:13.577009 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:13.577159 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:13.577215 (+ 56us) server_negotiation.cc:299] Negotiation successful
0812 09:45:13.577327 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":124,"threads_started":1}
I20260812 09:45:13.577787 16448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:13.571324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.3:38427 (local address 127.0.0.1:57466)
0812 09:45:13.572195 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:13.572207 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:13.572273 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:13.573368 (+ 1095us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:13.573371 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:13.573393 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:45:13.573506 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:13.573514 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:13.575092 (+ 1578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:13.575097 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:13.576265 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:13.576277 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:13.577380 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:13.577385 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:13.577509 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:13.577534 (+ 25us) client_negotiation.cc:769] Sending connection context
0812 09:45:13.577585 (+ 51us) client_negotiation.cc:240] Negotiation successful
0812 09:45:13.577650 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":801,"thread_start_us":96,"threads_started":1}
I20260812 09:45:13.578140 16451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:13.571395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51680 (local address 127.13.99.1:33687)
0812 09:45:13.572592 (+ 1197us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:13.572607 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:13.572633 (+ 26us) server_negotiation.cc:407] Connection header received
0812 09:45:13.572716 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:13.572721 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:13.572900 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:13.573076 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:45:13.573361 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:13.574212 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:13.575725 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:13.576610 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:13.576732 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:13.576890 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:13.577884 (+ 994us) server_negotiation.cc:299] Negotiation successful
0812 09:45:13.577990 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1051,"thread_start_us":90,"threads_started":1}
I20260812 09:45:13.578480 16456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:13.572165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57466 (local address 127.13.99.3:38427)
0812 09:45:13.572914 (+ 749us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:13.572926 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:13.572948 (+ 22us) server_negotiation.cc:407] Connection header received
0812 09:45:13.573022 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:13.573039 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:13.573209 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:13.573665 (+ 456us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:45:13.573803 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:13.574943 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:13.576401 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:13.577239 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:13.578012 (+ 773us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:13.578180 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:13.578234 (+ 54us) server_negotiation.cc:299] Negotiation successful
0812 09:45:13.578338 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":652,"thread_start_us":100,"threads_started":1}
I20260812 09:45:13.580737 16066 tablet_service.cc:1467] Processing CreateTablet for tablet 4c2367b510e04f8f917995471173046a (DEFAULT_TABLE table=test-table [id=f98375d1130c4fbf9b30c722f30eb0ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 09:45:13.580921 15831 tablet_service.cc:1467] Processing CreateTablet for tablet 4c2367b510e04f8f917995471173046a (DEFAULT_TABLE table=test-table [id=f98375d1130c4fbf9b30c722f30eb0ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 09:45:13.581116 16300 tablet_service.cc:1467] Processing CreateTablet for tablet 4c2367b510e04f8f917995471173046a (DEFAULT_TABLE table=test-table [id=f98375d1130c4fbf9b30c722f30eb0ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 09:45:13.581686 16066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c2367b510e04f8f917995471173046a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:45:13.581956 15831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c2367b510e04f8f917995471173046a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:45:13.581988 16300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c2367b510e04f8f917995471173046a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:45:13.588804 16457 tablet_bootstrap.cc:492] T 4c2367b510e04f8f917995471173046a P 0631d48e67d34bcb84ab4ecb65163865: Bootstrap starting.
I20260812 09:45:13.590188 16458 tablet_bootstrap.cc:492] T 4c2367b510e04f8f917995471173046a P 981793cd97244da8b9e2b369c1892d32: Bootstrap starting.
I20260812 09:45:13.591598 16457 tablet_bootstrap.cc:654] T 4c2367b510e04f8f917995471173046a P 0631d48e67d34bcb84ab4ecb65163865: Neither blocks nor log segments found. Creating new log.
I20260812 09:45:13.591892 16458 tablet_bootstrap.cc:654] T 4c2367b510e04f8f917995471173046a P 981793cd97244da8b9e2b369c1892d32: Neither blocks nor log segments found. Creating new log.
I20260812 09:45:13.592463 16457 log.cc:824] T 4c2367b510e04f8f917995471173046a P 0631d48e67d34bcb84ab4ecb65163865: Log is configured to *not* fsync() on all Append() calls
I20260812 09:45:13.593190 16461 tablet_bootstrap.cc:492] T 4c2367b510e04f8f917995471173046a P 54f7fe0d3da94da18132fb2d7b0ed0b8: Bootstrap starting.
I20260812 09:45:13.593242 16458 log.cc:824] T 4c2367b510e04f8f917995471173046a P 981793cd97244da8b9e2b369c1892d32: Log is configured to *not* fsync() on all Append() calls
I20260812 09:45:13.594622 16457 tablet_bootstrap.cc:492] T 4c2367b510e04f8f917995471173046a P 0631d48e67d34bcb84ab4ecb65163865: No bootstrap required, opened a new log
I20260812 09:45:13.594823 16457 ts_tablet_manager.cc:1397] T 4c2367b510e04f8f917995471173046a P 0631d48e67d34bcb84ab4ecb65163865: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20260812 09:45:13.594861 16458 tablet_bootstrap.cc:492] T 4c2367b510e04f8f917995471173046a P 981793cd97244da8b9e2b369c1892d32: No bootstrap required, opened a new log
I20260812 09:45:13.595068 16458 ts_tablet_manager.cc:1397] T 4c2367b510e04f8f917995471173046a P 981793cd97244da8b9e2b369c1892d32: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260812 09:45:13.595220 16461 tablet_bootstrap.cc:654] T 4c2367b510e04f8f917995471173046a P 54f7fe0d3da94da18132fb2d7b0ed0b8: Neither blocks nor log segments found. Creating new log.
I20260812 09:45:13.596031 16461 log.cc:824] T 4c2367b510e04f8f917995471173046a P 54f7fe0d3da94da18132fb2d7b0ed0b8: Log is configured to *not* fsync() on all Append() calls
I20260812 09:45:13.598189 16461 tablet_bootstrap.cc:492] T 4c2367b510e04f8f917995471173046a P 54f7fe0d3da94da18132fb2d7b0ed0b8: No bootstrap required, opened a new log
I20260812 09:45:13.598377 16461 ts_tablet_manager.cc:1397] T 4c2367b510e04f8f917995471173046a P 54f7fe0d3da94da18132fb2d7b0ed0b8: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260812 09:45:13.598573 16458 raft_consensus.cc:348] T 4c2367b510e04f8f917995471173046a P 981793cd97244da8b9e2b369c1892d32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "981793cd97244da8b9e2b369c1892d32" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 33687 } } peers { permanent_uuid: "54f7fe0d3da94da18132fb2d7b0ed0b8" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 38427 } } peers { permanent_uuid: "0631d48e67d34bcb84ab4ecb65163865" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35803 } }
I20260812 09:45:13.598805 16458 raft_consensus.cc:374] T 4c2367b510e04f8f917995471173046a P 981793cd97244da8b9e2b369c1892d32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:45:13.598868 16458 raft_consensus.cc:729] T 4c2367b510e04f8f917995471173046a P 981793cd97244da8b9e2b369c1892d32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 981793cd97244da8b9e2b369c1892d32, State: Initialized, Role: FOLLOWER
I20260812 09:45:13.598932 16457 raft_consensus.cc:348] T 4c2367b510e04f8f917995471173046a P 0631d48e67d34bcb84ab4ecb65163865 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "981793cd97244da8b9e2b369c1892d32" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 33687 } } peers { permanent_uuid: "54f7fe0d3da94da18132fb2d7b0ed0b8" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 38427 } } peers { permanent_uuid: "0631d48e67d34bcb84ab4ecb65163865" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35803 } }
I20260812 09:45:13.599193 16457 raft_consensus.cc:374] T 4c2367b510e04f8f917995471173046a P 0631d48e67d34bcb84ab4ecb65163865 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:45:13.599254 16457 raft_consensus.cc:729] T 4c2367b510e04f8f917995471173046a P 0631d48e67d34bcb84ab4ecb65163865 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0631d48e67d34bcb84ab4ecb65163865, State: Initialized, Role: FOLLOWER
I20260812 09:45:13.599321 16458 consensus_queue.cc:260] T 4c2367b510e04f8f917995471173046a P 981793cd97244da8b9e2b369c1892d32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "981793cd97244da8b9e2b369c1892d32" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 33687 } } peers { permanent_uuid: "54f7fe0d3da94da18132fb2d7b0ed0b8" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 38427 } } peers { permanent_uuid: "0631d48e67d34bcb84ab4ecb65163865" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35803 } }
I20260812 09:45:13.599700 16457 consensus_queue.cc:260] T 4c2367b510e04f8f917995471173046a P 0631d48e67d34bcb84ab4ecb65163865 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "981793cd97244da8b9e2b369c1892d32" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 33687 } } peers { permanent_uuid: "54f7fe0d3da94da18132fb2d7b0ed0b8" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 38427 } } peers { permanent_uuid: "0631d48e67d34bcb84ab4ecb65163865" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35803 } }
I20260812 09:45:13.600339 16458 ts_tablet_manager.cc:1428] T 4c2367b510e04f8f917995471173046a P 981793cd97244da8b9e2b369c1892d32: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260812 09:45:13.600733 15940 heartbeater.cc:503] Master 127.13.99.62:32955 was elected leader, sending a full tablet report...
I20260812 09:45:13.600737 16457 ts_tablet_manager.cc:1428] T 4c2367b510e04f8f917995471173046a P 0631d48e67d34bcb84ab4ecb65163865: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
I20260812 09:45:13.601032 16179 heartbeater.cc:503] Master 127.13.99.62:32955 was elected leader, sending a full tablet report...
I20260812 09:45:13.603073 16461 raft_consensus.cc:348] T 4c2367b510e04f8f917995471173046a P 54f7fe0d3da94da18132fb2d7b0ed0b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "981793cd97244da8b9e2b369c1892d32" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 33687 } } peers { permanent_uuid: "54f7fe0d3da94da18132fb2d7b0ed0b8" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 38427 } } peers { permanent_uuid: "0631d48e67d34bcb84ab4ecb65163865" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35803 } }
I20260812 09:45:13.603293 16461 raft_consensus.cc:374] T 4c2367b510e04f8f917995471173046a P 54f7fe0d3da94da18132fb2d7b0ed0b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:45:13.603369 16461 raft_consensus.cc:729] T 4c2367b510e04f8f917995471173046a P 54f7fe0d3da94da18132fb2d7b0ed0b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54f7fe0d3da94da18132fb2d7b0ed0b8, State: Initialized, Role: FOLLOWER
I20260812 09:45:13.603840 16461 consensus_queue.cc:260] T 4c2367b510e04f8f917995471173046a P 54f7fe0d3da94da18132fb2d7b0ed0b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "981793cd97244da8b9e2b369c1892d32" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 33687 } } peers { permanent_uuid: "54f7fe0d3da94da18132fb2d7b0ed0b8" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 38427 } } peers { permanent_uuid: "0631d48e67d34bcb84ab4ecb65163865" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35803 } }
I20260812 09:45:13.604452 16419 heartbeater.cc:503] Master 127.13.99.62:32955 was elected leader, sending a full tablet report...
I20260812 09:45:13.604839 16461 ts_tablet_manager.cc:1428] T 4c2367b510e04f8f917995471173046a P 54f7fe0d3da94da18132fb2d7b0ed0b8: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260812 09:45:13.699487 16465 raft_consensus.cc:482] T 4c2367b510e04f8f917995471173046a P 981793cd97244da8b9e2b369c1892d32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 09:45:13.699756 16465 raft_consensus.cc:504] T 4c2367b510e04f8f917995471173046a P 981793cd97244da8b9e2b369c1892d32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "981793cd97244da8b9e2b369c1892d32" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 33687 } } peers { permanent_uuid: "54f7fe0d3da94da18132fb2d7b0ed0b8" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 38427 } } peers { permanent_uuid: "0631d48e67d34bcb84ab4ecb65163865" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35803 } }
I20260812 09:45:13.701028 16465 leader_election.cc:288] T 4c2367b510e04f8f917995471173046a P 981793cd97244da8b9e2b369c1892d32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54f7fe0d3da94da18132fb2d7b0ed0b8 (127.13.99.3:38427), 0631d48e67d34bcb84ab4ecb65163865 (127.13.99.2:35803)
I20260812 09:45:13.705277 16456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:13.701538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.1:48805 (local address 127.13.99.3:38427)
0812 09:45:13.701765 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:13.701772 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:13.701878 (+ 106us) server_negotiation.cc:407] Connection header received
0812 09:45:13.702046 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:13.702049 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:13.702092 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:13.702186 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:45:13.702531 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:13.703036 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:13.704179 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:13.704618 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:13.704701 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:13.704791 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:13.705047 (+ 256us) server_negotiation.cc:299] Negotiation successful
0812 09:45:13.705129 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132}
I20260812 09:45:13.705282 16455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:13.701477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.1:54343 (local address 127.13.99.2:35803)
0812 09:45:13.701643 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:13.701648 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:13.701834 (+ 186us) server_negotiation.cc:407] Connection header received
0812 09:45:13.702023 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:13.702026 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:13.702083 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:13.702145 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:45:13.702505 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:13.703038 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:13.704180 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:13.704685 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:13.704799 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:13.704880 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:13.705090 (+ 210us) server_negotiation.cc:299] Negotiation successful
0812 09:45:13.705170 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260812 09:45:13.705313 16470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:13.701358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.3:38427 (local address 127.13.99.1:48805)
0812 09:45:13.701745 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:13.701758 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:13.701906 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:13.702224 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:13.702226 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:13.702273 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:45:13.702385 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:13.702390 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:13.703162 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:13.703168 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:13.704048 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:13.704057 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:13.704742 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:13.704749 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:13.704907 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:13.704932 (+ 25us) client_negotiation.cc:769] Sending connection context
0812 09:45:13.705002 (+ 70us) client_negotiation.cc:240] Negotiation successful
0812 09:45:13.705100 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"mutex_wait_us":40,"spinlock_wait_cycles":896,"thread_start_us":163,"threads_started":1}
I20260812 09:45:13.705319 16471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:13.701297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.2:35803 (local address 127.13.99.1:54343)
0812 09:45:13.701707 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:13.701723 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:13.701884 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:13.702189 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:13.702192 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:13.702209 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:45:13.702380 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:13.702386 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:13.703180 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:13.703184 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:13.704048 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:13.704058 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:13.704820 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:13.704824 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:13.704943 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:13.704961 (+ 18us) client_negotiation.cc:769] Sending connection context
0812 09:45:13.705058 (+ 97us) client_negotiation.cc:240] Negotiation successful
0812 09:45:13.705128 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"mutex_wait_us":42,"spinlock_wait_cycles":28544,"thread_start_us":153,"threads_started":1}
I20260812 09:45:13.706259 16335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c2367b510e04f8f917995471173046a" candidate_uuid: "981793cd97244da8b9e2b369c1892d32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54f7fe0d3da94da18132fb2d7b0ed0b8" is_pre_election: true
I20260812 09:45:13.706322 16106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c2367b510e04f8f917995471173046a" candidate_uuid: "981793cd97244da8b9e2b369c1892d32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0631d48e67d34bcb84ab4ecb65163865" is_pre_election: true
I20260812 09:45:13.706591 16335 raft_consensus.cc:2454] T 4c2367b510e04f8f917995471173046a P 54f7fe0d3da94da18132fb2d7b0ed0b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 981793cd97244da8b9e2b369c1892d32 in term 0.
I20260812 09:45:13.706651 16106 raft_consensus.cc:2454] T 4c2367b510e04f8f917995471173046a P 0631d48e67d34bcb84ab4ecb65163865 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 981793cd97244da8b9e2b369c1892d32 in term 0.
I20260812 09:45:13.707257 15737 leader_election.cc:302] T 4c2367b510e04f8f917995471173046a P 981793cd97244da8b9e2b369c1892d32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54f7fe0d3da94da18132fb2d7b0ed0b8, 981793cd97244da8b9e2b369c1892d32; no voters:
I20260812 09:45:13.707588 16465 raft_consensus.cc:2781] T 4c2367b510e04f8f917995471173046a P 981793cd97244da8b9e2b369c1892d32 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 09:45:13.707716 16465 raft_consensus.cc:482] T 4c2367b510e04f8f917995471173046a P 981793cd97244da8b9e2b369c1892d32 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 09:45:13.707773 16465 raft_consensus.cc:3037] T 4c2367b510e04f8f917995471173046a P 981793cd97244da8b9e2b369c1892d32 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:45:13.709203 16465 raft_consensus.cc:504] T 4c2367b510e04f8f917995471173046a P 981793cd97244da8b9e2b369c1892d32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "981793cd97244da8b9e2b369c1892d32" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 33687 } } peers { permanent_uuid: "54f7fe0d3da94da18132fb2d7b0ed0b8" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 38427 } } peers { permanent_uuid: "0631d48e67d34bcb84ab4ecb65163865" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35803 } }
I20260812 09:45:13.709649 16465 leader_election.cc:288] T 4c2367b510e04f8f917995471173046a P 981793cd97244da8b9e2b369c1892d32 [CANDIDATE]: Term 1 election: Requested vote from peers 54f7fe0d3da94da18132fb2d7b0ed0b8 (127.13.99.3:38427), 0631d48e67d34bcb84ab4ecb65163865 (127.13.99.2:35803)
I20260812 09:45:13.710039 16335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c2367b510e04f8f917995471173046a" candidate_uuid: "981793cd97244da8b9e2b369c1892d32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54f7fe0d3da94da18132fb2d7b0ed0b8"
I20260812 09:45:13.710201 16335 raft_consensus.cc:3037] T 4c2367b510e04f8f917995471173046a P 54f7fe0d3da94da18132fb2d7b0ed0b8 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:45:13.711938 16335 raft_consensus.cc:2454] T 4c2367b510e04f8f917995471173046a P 54f7fe0d3da94da18132fb2d7b0ed0b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 981793cd97244da8b9e2b369c1892d32 in term 1.
I20260812 09:45:13.712333 15737 leader_election.cc:302] T 4c2367b510e04f8f917995471173046a P 981793cd97244da8b9e2b369c1892d32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54f7fe0d3da94da18132fb2d7b0ed0b8, 981793cd97244da8b9e2b369c1892d32; no voters:
I20260812 09:45:13.712576 16465 raft_consensus.cc:2781] T 4c2367b510e04f8f917995471173046a P 981793cd97244da8b9e2b369c1892d32 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 09:45:13.712611 16106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c2367b510e04f8f917995471173046a" candidate_uuid: "981793cd97244da8b9e2b369c1892d32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0631d48e67d34bcb84ab4ecb65163865"
I20260812 09:45:13.712775 16106 raft_consensus.cc:3037] T 4c2367b510e04f8f917995471173046a P 0631d48e67d34bcb84ab4ecb65163865 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:45:13.712842 16465 raft_consensus.cc:686] T 4c2367b510e04f8f917995471173046a P 981793cd97244da8b9e2b369c1892d32 [term 1 LEADER]: Becoming Leader. State: Replica: 981793cd97244da8b9e2b369c1892d32, State: Running, Role: LEADER
I20260812 09:45:13.713239 16465 consensus_queue.cc:237] T 4c2367b510e04f8f917995471173046a P 981793cd97244da8b9e2b369c1892d32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "981793cd97244da8b9e2b369c1892d32" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 33687 } } peers { permanent_uuid: "54f7fe0d3da94da18132fb2d7b0ed0b8" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 38427 } } peers { permanent_uuid: "0631d48e67d34bcb84ab4ecb65163865" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35803 } }
I20260812 09:45:13.714617 16106 raft_consensus.cc:2454] T 4c2367b510e04f8f917995471173046a P 0631d48e67d34bcb84ab4ecb65163865 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 981793cd97244da8b9e2b369c1892d32 in term 1.
I20260812 09:45:13.717002 15587 catalog_manager.cc:5495] T 4c2367b510e04f8f917995471173046a P 981793cd97244da8b9e2b369c1892d32 reported cstate change: term changed from 0 to 1, leader changed from <none> to 981793cd97244da8b9e2b369c1892d32 (127.13.99.1). New cstate: current_term: 1 leader_uuid: "981793cd97244da8b9e2b369c1892d32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "981793cd97244da8b9e2b369c1892d32" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 33687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54f7fe0d3da94da18132fb2d7b0ed0b8" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 38427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0631d48e67d34bcb84ab4ecb65163865" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35803 } health_report { overall_health: UNKNOWN } } }
I20260812 09:45:13.741348 16451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:13.737310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51696 (local address 127.13.99.1:33687)
0812 09:45:13.737450 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:13.737454 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:13.737471 (+ 17us) server_negotiation.cc:407] Connection header received
0812 09:45:13.737753 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:13.737757 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:13.737825 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:13.737921 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 09:45:13.738215 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:13.738728 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:13.739669 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:13.739916 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:13.740009 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:13.740390 (+ 381us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 09:45:13.740973 (+ 583us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 09:45:13.741094 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:13.741170 (+ 76us) server_negotiation.cc:299] Negotiation successful
0812 09:45:13.741224 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260812 09:45:13.748413 16335 raft_consensus.cc:1261] T 4c2367b510e04f8f917995471173046a P 54f7fe0d3da94da18132fb2d7b0ed0b8 [term 1 FOLLOWER]: Refusing update from remote peer 981793cd97244da8b9e2b369c1892d32: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 09:45:13.749501 16473 consensus_queue.cc:1035] T 4c2367b510e04f8f917995471173046a P 981793cd97244da8b9e2b369c1892d32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54f7fe0d3da94da18132fb2d7b0ed0b8" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 38427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 09:45:13.749629 16106 raft_consensus.cc:1261] T 4c2367b510e04f8f917995471173046a P 0631d48e67d34bcb84ab4ecb65163865 [term 1 FOLLOWER]: Refusing update from remote peer 981793cd97244da8b9e2b369c1892d32: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 09:45:13.750321 16473 consensus_queue.cc:1035] T 4c2367b510e04f8f917995471173046a P 981793cd97244da8b9e2b369c1892d32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0631d48e67d34bcb84ab4ecb65163865" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 09:45:13.758654 16479 mvcc.cc:205] Tried to move back new op lower bound from 7317618334703538176 to 7317618334572351488. Current Snapshot: MvccSnapshot[applied={T|T < 7317618334703538176}]
I20260812 09:45:13.759500 16484 mvcc.cc:205] Tried to move back new op lower bound from 7317618334703538176 to 7317618334572351488. Current Snapshot: MvccSnapshot[applied={T|T < 7317618334703538176}]
I20260812 09:45:13.764371 16486 mvcc.cc:205] Tried to move back new op lower bound from 7317618334703538176 to 7317618334572351488. Current Snapshot: MvccSnapshot[applied={T|T < 7317618334703538176}]
I20260812 09:45:13.773015 15587 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46588:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 09:45:13.773303 15587 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46588:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 09:45:13.776710 15587 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3eac478794a6474684ce396fa014045f: Sending DeleteTablet for 3 replicas of tablet 4c2367b510e04f8f917995471173046a
I20260812 09:45:13.778141 15831 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c2367b510e04f8f917995471173046a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 09:45:13 UTC) from {username='kudu', principal='kudu/127.13.99.62@KRBTEST.COM'} at 127.0.0.1:51680
I20260812 09:45:13.778204 16300 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c2367b510e04f8f917995471173046a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 09:45:13 UTC) from {username='kudu', principal='kudu/127.13.99.62@KRBTEST.COM'} at 127.0.0.1:57466
I20260812 09:45:13.779206 15587 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46588:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 09:45:13.779680 15587 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 09:45:13.780234 16495 tablet_replica.cc:330] T 4c2367b510e04f8f917995471173046a P 981793cd97244da8b9e2b369c1892d32: stopping tablet replica
I20260812 09:45:13.780606 16066 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c2367b510e04f8f917995471173046a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 09:45:13 UTC) from {username='kudu', principal='kudu/127.13.99.62@KRBTEST.COM'} at 127.0.0.1:59222
I20260812 09:45:13.781134 16497 tablet_replica.cc:330] T 4c2367b510e04f8f917995471173046a P 0631d48e67d34bcb84ab4ecb65163865: stopping tablet replica
I20260812 09:45:13.781484 16497 raft_consensus.cc:2229] T 4c2367b510e04f8f917995471173046a P 0631d48e67d34bcb84ab4ecb65163865 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 09:45:13.781833 16497 raft_consensus.cc:2258] T 4c2367b510e04f8f917995471173046a P 0631d48e67d34bcb84ab4ecb65163865 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 09:45:13.782678 16495 raft_consensus.cc:2229] T 4c2367b510e04f8f917995471173046a P 981793cd97244da8b9e2b369c1892d32 [term 1 LEADER]: Raft consensus shutting down.
I20260812 09:45:13.784235 16497 ts_tablet_manager.cc:1905] T 4c2367b510e04f8f917995471173046a P 0631d48e67d34bcb84ab4ecb65163865: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 09:45:13.784710 16496 tablet_replica.cc:330] T 4c2367b510e04f8f917995471173046a P 54f7fe0d3da94da18132fb2d7b0ed0b8: stopping tablet replica
I20260812 09:45:13.785034 16496 raft_consensus.cc:2229] T 4c2367b510e04f8f917995471173046a P 54f7fe0d3da94da18132fb2d7b0ed0b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 09:45:13.786257 16496 raft_consensus.cc:2258] T 4c2367b510e04f8f917995471173046a P 54f7fe0d3da94da18132fb2d7b0ed0b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 09:45:13.788298 16300 tablet_service.cc:1467] Processing CreateTablet for tablet f1eac8933c884a32b98ef4c6627d4c4a (DEFAULT_TABLE table=test-table [id=3d0e63f262634573b99c60d1a195ba79]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 09:45:13.788599 16300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1eac8933c884a32b98ef4c6627d4c4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:45:13.789316 16495 raft_consensus.cc:2258] T 4c2367b510e04f8f917995471173046a P 981793cd97244da8b9e2b369c1892d32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 09:45:13.790601 15831 tablet_service.cc:1467] Processing CreateTablet for tablet f1eac8933c884a32b98ef4c6627d4c4a (DEFAULT_TABLE table=test-table [id=3d0e63f262634573b99c60d1a195ba79]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 09:45:13.790920 15831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1eac8933c884a32b98ef4c6627d4c4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:45:13.791445 16496 ts_tablet_manager.cc:1905] T 4c2367b510e04f8f917995471173046a P 54f7fe0d3da94da18132fb2d7b0ed0b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 09:45:13.793634 16066 tablet_service.cc:1467] Processing CreateTablet for tablet f1eac8933c884a32b98ef4c6627d4c4a (DEFAULT_TABLE table=test-table [id=3d0e63f262634573b99c60d1a195ba79]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 09:45:13.793901 16066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1eac8933c884a32b98ef4c6627d4c4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:45:13.793931 16497 ts_tablet_manager.cc:1918] T 4c2367b510e04f8f917995471173046a P 0631d48e67d34bcb84ab4ecb65163865: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 09:45:13.794086 16497 log.cc:1196] T 4c2367b510e04f8f917995471173046a P 0631d48e67d34bcb84ab4ecb65163865: Deleting WAL directory at /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/minicluster-data/ts-1/wal/wals/4c2367b510e04f8f917995471173046a
I20260812 09:45:13.795276 16497 ts_tablet_manager.cc:1939] T 4c2367b510e04f8f917995471173046a P 0631d48e67d34bcb84ab4ecb65163865: Deleting consensus metadata
I20260812 09:45:13.796080 16458 tablet_bootstrap.cc:492] T f1eac8933c884a32b98ef4c6627d4c4a P 981793cd97244da8b9e2b369c1892d32: Bootstrap starting.
I20260812 09:45:13.797201 16458 tablet_bootstrap.cc:654] T f1eac8933c884a32b98ef4c6627d4c4a P 981793cd97244da8b9e2b369c1892d32: Neither blocks nor log segments found. Creating new log.
I20260812 09:45:13.797914 16457 tablet_bootstrap.cc:492] T f1eac8933c884a32b98ef4c6627d4c4a P 0631d48e67d34bcb84ab4ecb65163865: Bootstrap starting.
I20260812 09:45:13.798816 16457 tablet_bootstrap.cc:654] T f1eac8933c884a32b98ef4c6627d4c4a P 0631d48e67d34bcb84ab4ecb65163865: Neither blocks nor log segments found. Creating new log.
I20260812 09:45:13.800249 15569 catalog_manager.cc:4841] TS 0631d48e67d34bcb84ab4ecb65163865 (127.13.99.2:35803): tablet 4c2367b510e04f8f917995471173046a (table test-table [id=f98375d1130c4fbf9b30c722f30eb0ca]) successfully deleted
I20260812 09:45:13.800766 16496 ts_tablet_manager.cc:1918] T 4c2367b510e04f8f917995471173046a P 54f7fe0d3da94da18132fb2d7b0ed0b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 09:45:13.800901 16496 log.cc:1196] T 4c2367b510e04f8f917995471173046a P 54f7fe0d3da94da18132fb2d7b0ed0b8: Deleting WAL directory at /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/minicluster-data/ts-2/wal/wals/4c2367b510e04f8f917995471173046a
I20260812 09:45:13.801258 16496 ts_tablet_manager.cc:1939] T 4c2367b510e04f8f917995471173046a P 54f7fe0d3da94da18132fb2d7b0ed0b8: Deleting consensus metadata
I20260812 09:45:13.801966 16461 tablet_bootstrap.cc:492] T f1eac8933c884a32b98ef4c6627d4c4a P 54f7fe0d3da94da18132fb2d7b0ed0b8: Bootstrap starting.
I20260812 09:45:13.802551 15568 catalog_manager.cc:4841] TS 54f7fe0d3da94da18132fb2d7b0ed0b8 (127.13.99.3:38427): tablet 4c2367b510e04f8f917995471173046a (table test-table [id=f98375d1130c4fbf9b30c722f30eb0ca]) successfully deleted
I20260812 09:45:13.802989 16461 tablet_bootstrap.cc:654] T f1eac8933c884a32b98ef4c6627d4c4a P 54f7fe0d3da94da18132fb2d7b0ed0b8: Neither blocks nor log segments found. Creating new log.
I20260812 09:45:13.804283 16461 tablet_bootstrap.cc:492] T f1eac8933c884a32b98ef4c6627d4c4a P 54f7fe0d3da94da18132fb2d7b0ed0b8: No bootstrap required, opened a new log
I20260812 09:45:13.804364 16461 ts_tablet_manager.cc:1397] T f1eac8933c884a32b98ef4c6627d4c4a P 54f7fe0d3da94da18132fb2d7b0ed0b8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 09:45:13.804913 16461 raft_consensus.cc:348] T f1eac8933c884a32b98ef4c6627d4c4a P 54f7fe0d3da94da18132fb2d7b0ed0b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "981793cd97244da8b9e2b369c1892d32" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 33687 } } peers { permanent_uuid: "0631d48e67d34bcb84ab4ecb65163865" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35803 } } peers { permanent_uuid: "54f7fe0d3da94da18132fb2d7b0ed0b8" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 38427 } }
I20260812 09:45:13.805050 16461 raft_consensus.cc:374] T f1eac8933c884a32b98ef4c6627d4c4a P 54f7fe0d3da94da18132fb2d7b0ed0b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:45:13.805089 16461 raft_consensus.cc:729] T f1eac8933c884a32b98ef4c6627d4c4a P 54f7fe0d3da94da18132fb2d7b0ed0b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54f7fe0d3da94da18132fb2d7b0ed0b8, State: Initialized, Role: FOLLOWER
I20260812 09:45:13.805229 16461 consensus_queue.cc:260] T f1eac8933c884a32b98ef4c6627d4c4a P 54f7fe0d3da94da18132fb2d7b0ed0b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "981793cd97244da8b9e2b369c1892d32" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 33687 } } peers { permanent_uuid: "0631d48e67d34bcb84ab4ecb65163865" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35803 } } peers { permanent_uuid: "54f7fe0d3da94da18132fb2d7b0ed0b8" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 38427 } }
I20260812 09:45:13.805521 16461 ts_tablet_manager.cc:1428] T f1eac8933c884a32b98ef4c6627d4c4a P 54f7fe0d3da94da18132fb2d7b0ed0b8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260812 09:45:13.813761 16495 ts_tablet_manager.cc:1905] T 4c2367b510e04f8f917995471173046a P 981793cd97244da8b9e2b369c1892d32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 09:45:13.816987 16495 ts_tablet_manager.cc:1918] T 4c2367b510e04f8f917995471173046a P 981793cd97244da8b9e2b369c1892d32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 09:45:13.819562 16457 tablet_bootstrap.cc:492] T f1eac8933c884a32b98ef4c6627d4c4a P 0631d48e67d34bcb84ab4ecb65163865: No bootstrap required, opened a new log
I20260812 09:45:13.819676 16457 ts_tablet_manager.cc:1397] T f1eac8933c884a32b98ef4c6627d4c4a P 0631d48e67d34bcb84ab4ecb65163865: Time spent bootstrapping tablet: real 0.022s user 0.001s sys 0.000s
I20260812 09:45:13.820233 16457 raft_consensus.cc:348] T f1eac8933c884a32b98ef4c6627d4c4a P 0631d48e67d34bcb84ab4ecb65163865 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "981793cd97244da8b9e2b369c1892d32" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 33687 } } peers { permanent_uuid: "0631d48e67d34bcb84ab4ecb65163865" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35803 } } peers { permanent_uuid: "54f7fe0d3da94da18132fb2d7b0ed0b8" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 38427 } }
I20260812 09:45:13.820377 16457 raft_consensus.cc:374] T f1eac8933c884a32b98ef4c6627d4c4a P 0631d48e67d34bcb84ab4ecb65163865 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:45:13.820425 16457 raft_consensus.cc:729] T f1eac8933c884a32b98ef4c6627d4c4a P 0631d48e67d34bcb84ab4ecb65163865 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0631d48e67d34bcb84ab4ecb65163865, State: Initialized, Role: FOLLOWER
I20260812 09:45:13.820618 16457 consensus_queue.cc:260] T f1eac8933c884a32b98ef4c6627d4c4a P 0631d48e67d34bcb84ab4ecb65163865 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "981793cd97244da8b9e2b369c1892d32" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 33687 } } peers { permanent_uuid: "0631d48e67d34bcb84ab4ecb65163865" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35803 } } peers { permanent_uuid: "54f7fe0d3da94da18132fb2d7b0ed0b8" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 38427 } }
I20260812 09:45:13.820957 16457 ts_tablet_manager.cc:1428] T f1eac8933c884a32b98ef4c6627d4c4a P 0631d48e67d34bcb84ab4ecb65163865: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 09:45:13.818305 16458 tablet_bootstrap.cc:492] T f1eac8933c884a32b98ef4c6627d4c4a P 981793cd97244da8b9e2b369c1892d32: No bootstrap required, opened a new log
I20260812 09:45:13.821153 16458 ts_tablet_manager.cc:1397] T f1eac8933c884a32b98ef4c6627d4c4a P 981793cd97244da8b9e2b369c1892d32: Time spent bootstrapping tablet: real 0.025s user 0.000s sys 0.002s
I20260812 09:45:13.821736 16458 raft_consensus.cc:348] T f1eac8933c884a32b98ef4c6627d4c4a P 981793cd97244da8b9e2b369c1892d32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "981793cd97244da8b9e2b369c1892d32" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 33687 } } peers { permanent_uuid: "0631d48e67d34bcb84ab4ecb65163865" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35803 } } peers { permanent_uuid: "54f7fe0d3da94da18132fb2d7b0ed0b8" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 38427 } }
I20260812 09:45:13.821858 16458 raft_consensus.cc:374] T f1eac8933c884a32b98ef4c6627d4c4a P 981793cd97244da8b9e2b369c1892d32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:45:13.821897 16458 raft_consensus.cc:729] T f1eac8933c884a32b98ef4c6627d4c4a P 981793cd97244da8b9e2b369c1892d32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 981793cd97244da8b9e2b369c1892d32, State: Initialized, Role: FOLLOWER
I20260812 09:45:13.822046 16458 consensus_queue.cc:260] T f1eac8933c884a32b98ef4c6627d4c4a P 981793cd97244da8b9e2b369c1892d32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "981793cd97244da8b9e2b369c1892d32" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 33687 } } peers { permanent_uuid: "0631d48e67d34bcb84ab4ecb65163865" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35803 } } peers { permanent_uuid: "54f7fe0d3da94da18132fb2d7b0ed0b8" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 38427 } }
I20260812 09:45:13.822331 16458 ts_tablet_manager.cc:1428] T f1eac8933c884a32b98ef4c6627d4c4a P 981793cd97244da8b9e2b369c1892d32: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260812 09:45:13.822484 16495 log.cc:1196] T 4c2367b510e04f8f917995471173046a P 981793cd97244da8b9e2b369c1892d32: Deleting WAL directory at /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/minicluster-data/ts-0/wal/wals/4c2367b510e04f8f917995471173046a
I20260812 09:45:13.822852 16495 ts_tablet_manager.cc:1939] T 4c2367b510e04f8f917995471173046a P 981793cd97244da8b9e2b369c1892d32: Deleting consensus metadata
I20260812 09:45:13.824182 15567 catalog_manager.cc:4841] TS 981793cd97244da8b9e2b369c1892d32 (127.13.99.1:33687): tablet 4c2367b510e04f8f917995471173046a (table test-table [id=f98375d1130c4fbf9b30c722f30eb0ca]) successfully deleted
W20260812 09:45:14.023963 16420 tablet.cc:2369] T f1eac8933c884a32b98ef4c6627d4c4a P 54f7fe0d3da94da18132fb2d7b0ed0b8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 09:45:14.041932 16180 tablet.cc:2369] T f1eac8933c884a32b98ef4c6627d4c4a P 0631d48e67d34bcb84ab4ecb65163865: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 09:45:14.072655 15942 tablet.cc:2369] T f1eac8933c884a32b98ef4c6627d4c4a P 981793cd97244da8b9e2b369c1892d32: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 09:45:14.103683 16466 raft_consensus.cc:482] T f1eac8933c884a32b98ef4c6627d4c4a P 0631d48e67d34bcb84ab4ecb65163865 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 09:45:14.103880 16466 raft_consensus.cc:504] T f1eac8933c884a32b98ef4c6627d4c4a P 0631d48e67d34bcb84ab4ecb65163865 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "981793cd97244da8b9e2b369c1892d32" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 33687 } } peers { permanent_uuid: "0631d48e67d34bcb84ab4ecb65163865" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35803 } } peers { permanent_uuid: "54f7fe0d3da94da18132fb2d7b0ed0b8" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 38427 } }
I20260812 09:45:14.105127 16466 leader_election.cc:288] T f1eac8933c884a32b98ef4c6627d4c4a P 0631d48e67d34bcb84ab4ecb65163865 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 981793cd97244da8b9e2b369c1892d32 (127.13.99.1:33687), 54f7fe0d3da94da18132fb2d7b0ed0b8 (127.13.99.3:38427)
I20260812 09:45:14.109421 16456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:14.105524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.2:54519 (local address 127.13.99.3:38427)
0812 09:45:14.105672 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:14.105677 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:14.105966 (+ 289us) server_negotiation.cc:407] Connection header received
0812 09:45:14.106126 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:14.106129 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:14.106178 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:14.106267 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:45:14.106537 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:14.107040 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:14.108329 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:14.108837 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:14.108978 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:14.109044 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:14.109227 (+ 183us) server_negotiation.cc:299] Negotiation successful
0812 09:45:14.109297 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260812 09:45:14.109449 16503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:14.105409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.3:38427 (local address 127.13.99.2:54519)
0812 09:45:14.105861 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:14.105877 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:14.106010 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:14.106281 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:14.106284 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:14.106300 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:45:14.106425 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:14.106431 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:14.107153 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:14.107157 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:14.108177 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:14.108187 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:14.108984 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:14.108988 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:14.109088 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:14.109118 (+ 30us) client_negotiation.cc:769] Sending connection context
0812 09:45:14.109209 (+ 91us) client_negotiation.cc:240] Negotiation successful
0812 09:45:14.109279 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"mutex_wait_us":133,"thread_start_us":82,"threads_started":1}
I20260812 09:45:14.109597 16451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:14.105705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.2:38971 (local address 127.13.99.1:33687)
0812 09:45:14.105881 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:14.105886 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:14.105902 (+ 16us) server_negotiation.cc:407] Connection header received
0812 09:45:14.106050 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:14.106053 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:14.106095 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:14.106184 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:45:14.106458 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:14.107003 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:14.108406 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:14.108906 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:14.109082 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:14.109147 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:14.109420 (+ 273us) server_negotiation.cc:299] Negotiation successful
0812 09:45:14.109491 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260812 09:45:14.109615 16506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:14.105409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.1:33687 (local address 127.13.99.2:38971)
0812 09:45:14.105784 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:14.105801 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:14.105917 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:14.106192 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:14.106196 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:14.106211 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:45:14.106347 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:14.106353 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:14.107121 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:14.107126 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:14.108221 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:14.108232 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:14.109216 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:14.109218 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:14.109287 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:14.109305 (+ 18us) client_negotiation.cc:769] Sending connection context
0812 09:45:14.109399 (+ 94us) client_negotiation.cc:240] Negotiation successful
0812 09:45:14.109466 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"mutex_wait_us":2,"thread_start_us":79,"threads_started":1}
I20260812 09:45:14.109886 16335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1eac8933c884a32b98ef4c6627d4c4a" candidate_uuid: "0631d48e67d34bcb84ab4ecb65163865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54f7fe0d3da94da18132fb2d7b0ed0b8" is_pre_election: true
I20260812 09:45:14.110066 16335 raft_consensus.cc:2454] T f1eac8933c884a32b98ef4c6627d4c4a P 54f7fe0d3da94da18132fb2d7b0ed0b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0631d48e67d34bcb84ab4ecb65163865 in term 0.
I20260812 09:45:14.110193 15864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1eac8933c884a32b98ef4c6627d4c4a" candidate_uuid: "0631d48e67d34bcb84ab4ecb65163865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "981793cd97244da8b9e2b369c1892d32" is_pre_election: true
I20260812 09:45:14.110455 15864 raft_consensus.cc:2454] T f1eac8933c884a32b98ef4c6627d4c4a P 981793cd97244da8b9e2b369c1892d32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0631d48e67d34bcb84ab4ecb65163865 in term 0.
I20260812 09:45:14.110469 15995 leader_election.cc:302] T f1eac8933c884a32b98ef4c6627d4c4a P 0631d48e67d34bcb84ab4ecb65163865 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0631d48e67d34bcb84ab4ecb65163865, 54f7fe0d3da94da18132fb2d7b0ed0b8; no voters:
I20260812 09:45:14.110750 16466 raft_consensus.cc:2781] T f1eac8933c884a32b98ef4c6627d4c4a P 0631d48e67d34bcb84ab4ecb65163865 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 09:45:14.110850 16466 raft_consensus.cc:482] T f1eac8933c884a32b98ef4c6627d4c4a P 0631d48e67d34bcb84ab4ecb65163865 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 09:45:14.110910 16466 raft_consensus.cc:3037] T f1eac8933c884a32b98ef4c6627d4c4a P 0631d48e67d34bcb84ab4ecb65163865 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:45:14.111918 16466 raft_consensus.cc:504] T f1eac8933c884a32b98ef4c6627d4c4a P 0631d48e67d34bcb84ab4ecb65163865 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "981793cd97244da8b9e2b369c1892d32" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 33687 } } peers { permanent_uuid: "0631d48e67d34bcb84ab4ecb65163865" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35803 } } peers { permanent_uuid: "54f7fe0d3da94da18132fb2d7b0ed0b8" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 38427 } }
I20260812 09:45:14.112289 16466 leader_election.cc:288] T f1eac8933c884a32b98ef4c6627d4c4a P 0631d48e67d34bcb84ab4ecb65163865 [CANDIDATE]: Term 1 election: Requested vote from peers 981793cd97244da8b9e2b369c1892d32 (127.13.99.1:33687), 54f7fe0d3da94da18132fb2d7b0ed0b8 (127.13.99.3:38427)
I20260812 09:45:14.112674 15864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1eac8933c884a32b98ef4c6627d4c4a" candidate_uuid: "0631d48e67d34bcb84ab4ecb65163865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "981793cd97244da8b9e2b369c1892d32"
I20260812 09:45:14.112674 16335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1eac8933c884a32b98ef4c6627d4c4a" candidate_uuid: "0631d48e67d34bcb84ab4ecb65163865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54f7fe0d3da94da18132fb2d7b0ed0b8"
I20260812 09:45:14.112798 15864 raft_consensus.cc:3037] T f1eac8933c884a32b98ef4c6627d4c4a P 981793cd97244da8b9e2b369c1892d32 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:45:14.112808 16335 raft_consensus.cc:3037] T f1eac8933c884a32b98ef4c6627d4c4a P 54f7fe0d3da94da18132fb2d7b0ed0b8 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:45:14.113982 15864 raft_consensus.cc:2454] T f1eac8933c884a32b98ef4c6627d4c4a P 981793cd97244da8b9e2b369c1892d32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0631d48e67d34bcb84ab4ecb65163865 in term 1.
I20260812 09:45:14.113982 16335 raft_consensus.cc:2454] T f1eac8933c884a32b98ef4c6627d4c4a P 54f7fe0d3da94da18132fb2d7b0ed0b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0631d48e67d34bcb84ab4ecb65163865 in term 1.
I20260812 09:45:14.114256 15995 leader_election.cc:302] T f1eac8933c884a32b98ef4c6627d4c4a P 0631d48e67d34bcb84ab4ecb65163865 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0631d48e67d34bcb84ab4ecb65163865, 54f7fe0d3da94da18132fb2d7b0ed0b8; no voters:
I20260812 09:45:14.114439 16466 raft_consensus.cc:2781] T f1eac8933c884a32b98ef4c6627d4c4a P 0631d48e67d34bcb84ab4ecb65163865 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 09:45:14.114621 16466 raft_consensus.cc:686] T f1eac8933c884a32b98ef4c6627d4c4a P 0631d48e67d34bcb84ab4ecb65163865 [term 1 LEADER]: Becoming Leader. State: Replica: 0631d48e67d34bcb84ab4ecb65163865, State: Running, Role: LEADER
I20260812 09:45:14.114902 16466 consensus_queue.cc:237] T f1eac8933c884a32b98ef4c6627d4c4a P 0631d48e67d34bcb84ab4ecb65163865 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "981793cd97244da8b9e2b369c1892d32" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 33687 } } peers { permanent_uuid: "0631d48e67d34bcb84ab4ecb65163865" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35803 } } peers { permanent_uuid: "54f7fe0d3da94da18132fb2d7b0ed0b8" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 38427 } }
I20260812 09:45:14.117205 15587 catalog_manager.cc:5495] T f1eac8933c884a32b98ef4c6627d4c4a P 0631d48e67d34bcb84ab4ecb65163865 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0631d48e67d34bcb84ab4ecb65163865 (127.13.99.2). New cstate: current_term: 1 leader_uuid: "0631d48e67d34bcb84ab4ecb65163865" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "981793cd97244da8b9e2b369c1892d32" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 33687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0631d48e67d34bcb84ab4ecb65163865" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54f7fe0d3da94da18132fb2d7b0ed0b8" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 38427 } health_report { overall_health: UNKNOWN } } }
Aug 12 09:45:14 dist-test-slave-jq4t krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527912, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.62@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
I20260812 09:45:14.187032 16521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:14.178295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46592 (local address 127.13.99.62:32955)
0812 09:45:14.178605 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:14.178613 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:14.178680 (+ 67us) server_negotiation.cc:407] Connection header received
0812 09:45:14.178792 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:14.178795 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:14.178835 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:14.178914 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:14.179606 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:14.180168 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:14.181000 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:14.181241 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:14.181321 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:14.183804 (+ 2483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:14.183807 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:14.183810 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:14.183832 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:14.185390 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:14.185834 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:14.185836 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:14.185837 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:14.185919 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:14.186201 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:14.186204 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:14.186205 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:14.186400 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:14.186485 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:14.186770 (+ 285us) server_negotiation.cc:299] Negotiation successful
0812 09:45:14.186881 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":135,"threads_started":1}
I20260812 09:45:14.187017 16520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:14.178240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:32955 (local address 127.0.0.1:46592)
0812 09:45:14.178587 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:14.178601 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:14.178709 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:14.178963 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:14.178965 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:14.179391 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:45:14.179488 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:14.179494 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:14.180278 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:14.180281 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:14.180894 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:14.180903 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:14.181382 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:14.181385 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:14.181467 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:14.182139 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:45:14.182151 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:45:14.183657 (+ 1506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:45:14.185493 (+ 1836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:14.185502 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:14.185508 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:14.185738 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:14.186009 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:14.186011 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:14.186012 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:14.186093 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:14.186502 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:45:14.186507 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:45:14.186615 (+ 108us) client_negotiation.cc:769] Sending connection context
0812 09:45:14.186709 (+ 94us) client_negotiation.cc:240] Negotiation successful
0812 09:45:14.186820 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":97,"threads_started":1}
I20260812 09:45:14.191516 15587 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.13.99.62@KRBTEST.COM'} at 127.0.0.1:46592:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260812 09:45:14.192299 15587 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 09:45:14.197522 15831 tablet_service.cc:1467] Processing CreateTablet for tablet 048b56eb17fc421c845e82b2e7546f61 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=092995521cb9449da73b1cfd767bf337]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 09:45:14.197686 16300 tablet_service.cc:1467] Processing CreateTablet for tablet 048b56eb17fc421c845e82b2e7546f61 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=092995521cb9449da73b1cfd767bf337]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 09:45:14.197985 15831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 048b56eb17fc421c845e82b2e7546f61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:45:14.198122 16300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 048b56eb17fc421c845e82b2e7546f61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:45:14.197544 16066 tablet_service.cc:1467] Processing CreateTablet for tablet 048b56eb17fc421c845e82b2e7546f61 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=092995521cb9449da73b1cfd767bf337]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 09:45:14.200181 16066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 048b56eb17fc421c845e82b2e7546f61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:45:14.200542 16458 tablet_bootstrap.cc:492] T 048b56eb17fc421c845e82b2e7546f61 P 981793cd97244da8b9e2b369c1892d32: Bootstrap starting.
I20260812 09:45:14.200774 16461 tablet_bootstrap.cc:492] T 048b56eb17fc421c845e82b2e7546f61 P 54f7fe0d3da94da18132fb2d7b0ed0b8: Bootstrap starting.
I20260812 09:45:14.201445 16458 tablet_bootstrap.cc:654] T 048b56eb17fc421c845e82b2e7546f61 P 981793cd97244da8b9e2b369c1892d32: Neither blocks nor log segments found. Creating new log.
I20260812 09:45:14.201745 16461 tablet_bootstrap.cc:654] T 048b56eb17fc421c845e82b2e7546f61 P 54f7fe0d3da94da18132fb2d7b0ed0b8: Neither blocks nor log segments found. Creating new log.
I20260812 09:45:14.202731 16458 tablet_bootstrap.cc:492] T 048b56eb17fc421c845e82b2e7546f61 P 981793cd97244da8b9e2b369c1892d32: No bootstrap required, opened a new log
I20260812 09:45:14.202824 16458 ts_tablet_manager.cc:1397] T 048b56eb17fc421c845e82b2e7546f61 P 981793cd97244da8b9e2b369c1892d32: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 09:45:14.203245 16458 raft_consensus.cc:348] T 048b56eb17fc421c845e82b2e7546f61 P 981793cd97244da8b9e2b369c1892d32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "981793cd97244da8b9e2b369c1892d32" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 33687 } } peers { permanent_uuid: "0631d48e67d34bcb84ab4ecb65163865" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35803 } } peers { permanent_uuid: "54f7fe0d3da94da18132fb2d7b0ed0b8" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 38427 } }
I20260812 09:45:14.203397 16458 raft_consensus.cc:374] T 048b56eb17fc421c845e82b2e7546f61 P 981793cd97244da8b9e2b369c1892d32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:45:14.203439 16458 raft_consensus.cc:729] T 048b56eb17fc421c845e82b2e7546f61 P 981793cd97244da8b9e2b369c1892d32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 981793cd97244da8b9e2b369c1892d32, State: Initialized, Role: FOLLOWER
I20260812 09:45:14.203598 16458 consensus_queue.cc:260] T 048b56eb17fc421c845e82b2e7546f61 P 981793cd97244da8b9e2b369c1892d32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "981793cd97244da8b9e2b369c1892d32" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 33687 } } peers { permanent_uuid: "0631d48e67d34bcb84ab4ecb65163865" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35803 } } peers { permanent_uuid: "54f7fe0d3da94da18132fb2d7b0ed0b8" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 38427 } }
I20260812 09:45:14.203946 16491 tablet_replica.cc:439] T 048b56eb17fc421c845e82b2e7546f61 P 981793cd97244da8b9e2b369c1892d32: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "981793cd97244da8b9e2b369c1892d32" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 33687 } } peers { permanent_uuid: "0631d48e67d34bcb84ab4ecb65163865" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35803 } } peers { permanent_uuid: "54f7fe0d3da94da18132fb2d7b0ed0b8" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 38427 } } }
I20260812 09:45:14.204087 16491 tablet_replica.cc:442] T 048b56eb17fc421c845e82b2e7546f61 P 981793cd97244da8b9e2b369c1892d32: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 09:45:14.204353 16458 ts_tablet_manager.cc:1428] T 048b56eb17fc421c845e82b2e7546f61 P 981793cd97244da8b9e2b369c1892d32: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 09:45:14.205313 16457 tablet_bootstrap.cc:492] T 048b56eb17fc421c845e82b2e7546f61 P 0631d48e67d34bcb84ab4ecb65163865: Bootstrap starting.
I20260812 09:45:14.206254 16461 tablet_bootstrap.cc:492] T 048b56eb17fc421c845e82b2e7546f61 P 54f7fe0d3da94da18132fb2d7b0ed0b8: No bootstrap required, opened a new log
I20260812 09:45:14.206374 16461 ts_tablet_manager.cc:1397] T 048b56eb17fc421c845e82b2e7546f61 P 54f7fe0d3da94da18132fb2d7b0ed0b8: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20260812 09:45:14.206460 16457 tablet_bootstrap.cc:654] T 048b56eb17fc421c845e82b2e7546f61 P 0631d48e67d34bcb84ab4ecb65163865: Neither blocks nor log segments found. Creating new log.
I20260812 09:45:14.206863 16461 raft_consensus.cc:348] T 048b56eb17fc421c845e82b2e7546f61 P 54f7fe0d3da94da18132fb2d7b0ed0b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "981793cd97244da8b9e2b369c1892d32" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 33687 } } peers { permanent_uuid: "0631d48e67d34bcb84ab4ecb65163865" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35803 } } peers { permanent_uuid: "54f7fe0d3da94da18132fb2d7b0ed0b8" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 38427 } }
I20260812 09:45:14.206982 16461 raft_consensus.cc:374] T 048b56eb17fc421c845e82b2e7546f61 P 54f7fe0d3da94da18132fb2d7b0ed0b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:45:14.207023 16461 raft_consensus.cc:729] T 048b56eb17fc421c845e82b2e7546f61 P 54f7fe0d3da94da18132fb2d7b0ed0b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54f7fe0d3da94da18132fb2d7b0ed0b8, State: Initialized, Role: FOLLOWER
I20260812 09:45:14.207172 16461 consensus_queue.cc:260] T 048b56eb17fc421c845e82b2e7546f61 P 54f7fe0d3da94da18132fb2d7b0ed0b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "981793cd97244da8b9e2b369c1892d32" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 33687 } } peers { permanent_uuid: "0631d48e67d34bcb84ab4ecb65163865" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35803 } } peers { permanent_uuid: "54f7fe0d3da94da18132fb2d7b0ed0b8" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 38427 } }
I20260812 09:45:14.207436 16469 tablet_replica.cc:439] T 048b56eb17fc421c845e82b2e7546f61 P 54f7fe0d3da94da18132fb2d7b0ed0b8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "981793cd97244da8b9e2b369c1892d32" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 33687 } } peers { permanent_uuid: "0631d48e67d34bcb84ab4ecb65163865" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35803 } } peers { permanent_uuid: "54f7fe0d3da94da18132fb2d7b0ed0b8" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 38427 } } }
I20260812 09:45:14.207528 16461 ts_tablet_manager.cc:1428] T 048b56eb17fc421c845e82b2e7546f61 P 54f7fe0d3da94da18132fb2d7b0ed0b8: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260812 09:45:14.207605 16469 tablet_replica.cc:442] T 048b56eb17fc421c845e82b2e7546f61 P 54f7fe0d3da94da18132fb2d7b0ed0b8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 09:45:14.208540 16457 tablet_bootstrap.cc:492] T 048b56eb17fc421c845e82b2e7546f61 P 0631d48e67d34bcb84ab4ecb65163865: No bootstrap required, opened a new log
I20260812 09:45:14.208647 16457 ts_tablet_manager.cc:1397] T 048b56eb17fc421c845e82b2e7546f61 P 0631d48e67d34bcb84ab4ecb65163865: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 09:45:14.209092 16457 raft_consensus.cc:348] T 048b56eb17fc421c845e82b2e7546f61 P 0631d48e67d34bcb84ab4ecb65163865 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "981793cd97244da8b9e2b369c1892d32" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 33687 } } peers { permanent_uuid: "0631d48e67d34bcb84ab4ecb65163865" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35803 } } peers { permanent_uuid: "54f7fe0d3da94da18132fb2d7b0ed0b8" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 38427 } }
I20260812 09:45:14.209230 16457 raft_consensus.cc:374] T 048b56eb17fc421c845e82b2e7546f61 P 0631d48e67d34bcb84ab4ecb65163865 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:45:14.209298 16457 raft_consensus.cc:729] T 048b56eb17fc421c845e82b2e7546f61 P 0631d48e67d34bcb84ab4ecb65163865 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0631d48e67d34bcb84ab4ecb65163865, State: Initialized, Role: FOLLOWER
I20260812 09:45:14.209463 16457 consensus_queue.cc:260] T 048b56eb17fc421c845e82b2e7546f61 P 0631d48e67d34bcb84ab4ecb65163865 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "981793cd97244da8b9e2b369c1892d32" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 33687 } } peers { permanent_uuid: "0631d48e67d34bcb84ab4ecb65163865" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35803 } } peers { permanent_uuid: "54f7fe0d3da94da18132fb2d7b0ed0b8" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 38427 } }
I20260812 09:45:14.209710 16466 tablet_replica.cc:439] T 048b56eb17fc421c845e82b2e7546f61 P 0631d48e67d34bcb84ab4ecb65163865: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "981793cd97244da8b9e2b369c1892d32" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 33687 } } peers { permanent_uuid: "0631d48e67d34bcb84ab4ecb65163865" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35803 } } peers { permanent_uuid: "54f7fe0d3da94da18132fb2d7b0ed0b8" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 38427 } } }
I20260812 09:45:14.209846 16466 tablet_replica.cc:442] T 048b56eb17fc421c845e82b2e7546f61 P 0631d48e67d34bcb84ab4ecb65163865: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 09:45:14.210110 16457 ts_tablet_manager.cc:1428] T 048b56eb17fc421c845e82b2e7546f61 P 0631d48e67d34bcb84ab4ecb65163865: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 09:45:14.337953 16466 raft_consensus.cc:482] T 048b56eb17fc421c845e82b2e7546f61 P 0631d48e67d34bcb84ab4ecb65163865 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 09:45:14.338083 16466 raft_consensus.cc:504] T 048b56eb17fc421c845e82b2e7546f61 P 0631d48e67d34bcb84ab4ecb65163865 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "981793cd97244da8b9e2b369c1892d32" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 33687 } } peers { permanent_uuid: "0631d48e67d34bcb84ab4ecb65163865" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35803 } } peers { permanent_uuid: "54f7fe0d3da94da18132fb2d7b0ed0b8" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 38427 } }
I20260812 09:45:14.338459 16466 leader_election.cc:288] T 048b56eb17fc421c845e82b2e7546f61 P 0631d48e67d34bcb84ab4ecb65163865 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 981793cd97244da8b9e2b369c1892d32 (127.13.99.1:33687), 54f7fe0d3da94da18132fb2d7b0ed0b8 (127.13.99.3:38427)
I20260812 09:45:14.338968 16335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "048b56eb17fc421c845e82b2e7546f61" candidate_uuid: "0631d48e67d34bcb84ab4ecb65163865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54f7fe0d3da94da18132fb2d7b0ed0b8" is_pre_election: true
I20260812 09:45:14.338968 15864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "048b56eb17fc421c845e82b2e7546f61" candidate_uuid: "0631d48e67d34bcb84ab4ecb65163865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "981793cd97244da8b9e2b369c1892d32" is_pre_election: true
I20260812 09:45:14.339138 15864 raft_consensus.cc:2454] T 048b56eb17fc421c845e82b2e7546f61 P 981793cd97244da8b9e2b369c1892d32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0631d48e67d34bcb84ab4ecb65163865 in term 0.
I20260812 09:45:14.339138 16335 raft_consensus.cc:2454] T 048b56eb17fc421c845e82b2e7546f61 P 54f7fe0d3da94da18132fb2d7b0ed0b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0631d48e67d34bcb84ab4ecb65163865 in term 0.
I20260812 09:45:14.339465 15995 leader_election.cc:302] T 048b56eb17fc421c845e82b2e7546f61 P 0631d48e67d34bcb84ab4ecb65163865 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0631d48e67d34bcb84ab4ecb65163865, 54f7fe0d3da94da18132fb2d7b0ed0b8; no voters:
I20260812 09:45:14.339669 16466 raft_consensus.cc:2781] T 048b56eb17fc421c845e82b2e7546f61 P 0631d48e67d34bcb84ab4ecb65163865 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 09:45:14.339732 16466 raft_consensus.cc:482] T 048b56eb17fc421c845e82b2e7546f61 P 0631d48e67d34bcb84ab4ecb65163865 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 09:45:14.339759 16466 raft_consensus.cc:3037] T 048b56eb17fc421c845e82b2e7546f61 P 0631d48e67d34bcb84ab4ecb65163865 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:45:14.340807 16466 raft_consensus.cc:504] T 048b56eb17fc421c845e82b2e7546f61 P 0631d48e67d34bcb84ab4ecb65163865 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "981793cd97244da8b9e2b369c1892d32" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 33687 } } peers { permanent_uuid: "0631d48e67d34bcb84ab4ecb65163865" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35803 } } peers { permanent_uuid: "54f7fe0d3da94da18132fb2d7b0ed0b8" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 38427 } }
I20260812 09:45:14.341154 16466 leader_election.cc:288] T 048b56eb17fc421c845e82b2e7546f61 P 0631d48e67d34bcb84ab4ecb65163865 [CANDIDATE]: Term 1 election: Requested vote from peers 981793cd97244da8b9e2b369c1892d32 (127.13.99.1:33687), 54f7fe0d3da94da18132fb2d7b0ed0b8 (127.13.99.3:38427)
I20260812 09:45:14.341444 15864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "048b56eb17fc421c845e82b2e7546f61" candidate_uuid: "0631d48e67d34bcb84ab4ecb65163865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "981793cd97244da8b9e2b369c1892d32"
I20260812 09:45:14.341601 15864 raft_consensus.cc:3037] T 048b56eb17fc421c845e82b2e7546f61 P 981793cd97244da8b9e2b369c1892d32 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:45:14.341563 16335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "048b56eb17fc421c845e82b2e7546f61" candidate_uuid: "0631d48e67d34bcb84ab4ecb65163865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54f7fe0d3da94da18132fb2d7b0ed0b8"
I20260812 09:45:14.341686 16335 raft_consensus.cc:3037] T 048b56eb17fc421c845e82b2e7546f61 P 54f7fe0d3da94da18132fb2d7b0ed0b8 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:45:14.342772 15864 raft_consensus.cc:2454] T 048b56eb17fc421c845e82b2e7546f61 P 981793cd97244da8b9e2b369c1892d32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0631d48e67d34bcb84ab4ecb65163865 in term 1.
I20260812 09:45:14.342772 16335 raft_consensus.cc:2454] T 048b56eb17fc421c845e82b2e7546f61 P 54f7fe0d3da94da18132fb2d7b0ed0b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0631d48e67d34bcb84ab4ecb65163865 in term 1.
I20260812 09:45:14.343075 15995 leader_election.cc:302] T 048b56eb17fc421c845e82b2e7546f61 P 0631d48e67d34bcb84ab4ecb65163865 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0631d48e67d34bcb84ab4ecb65163865, 54f7fe0d3da94da18132fb2d7b0ed0b8; no voters:
I20260812 09:45:14.343274 16466 raft_consensus.cc:2781] T 048b56eb17fc421c845e82b2e7546f61 P 0631d48e67d34bcb84ab4ecb65163865 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 09:45:14.343369 16466 raft_consensus.cc:686] T 048b56eb17fc421c845e82b2e7546f61 P 0631d48e67d34bcb84ab4ecb65163865 [term 1 LEADER]: Becoming Leader. State: Replica: 0631d48e67d34bcb84ab4ecb65163865, State: Running, Role: LEADER
I20260812 09:45:14.343506 16466 consensus_queue.cc:237] T 048b56eb17fc421c845e82b2e7546f61 P 0631d48e67d34bcb84ab4ecb65163865 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "981793cd97244da8b9e2b369c1892d32" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 33687 } } peers { permanent_uuid: "0631d48e67d34bcb84ab4ecb65163865" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35803 } } peers { permanent_uuid: "54f7fe0d3da94da18132fb2d7b0ed0b8" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 38427 } }
I20260812 09:45:14.344197 16511 tablet_replica.cc:439] T 048b56eb17fc421c845e82b2e7546f61 P 0631d48e67d34bcb84ab4ecb65163865: TxnStatusTablet state changed. Reason: New leader 0631d48e67d34bcb84ab4ecb65163865. Latest consensus state: current_term: 1 leader_uuid: "0631d48e67d34bcb84ab4ecb65163865" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "981793cd97244da8b9e2b369c1892d32" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 33687 } } peers { permanent_uuid: "0631d48e67d34bcb84ab4ecb65163865" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35803 } } peers { permanent_uuid: "54f7fe0d3da94da18132fb2d7b0ed0b8" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 38427 } } }
I20260812 09:45:14.344318 16511 tablet_replica.cc:442] T 048b56eb17fc421c845e82b2e7546f61 P 0631d48e67d34bcb84ab4ecb65163865: This TxnStatusTablet replica's current role is: LEADER
I20260812 09:45:14.344660 16530 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260812 09:45:14.344906 16530 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260812 09:45:14.345355 15587 catalog_manager.cc:5495] T 048b56eb17fc421c845e82b2e7546f61 P 0631d48e67d34bcb84ab4ecb65163865 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0631d48e67d34bcb84ab4ecb65163865 (127.13.99.2). New cstate: current_term: 1 leader_uuid: "0631d48e67d34bcb84ab4ecb65163865" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "981793cd97244da8b9e2b369c1892d32" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 33687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0631d48e67d34bcb84ab4ecb65163865" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54f7fe0d3da94da18132fb2d7b0ed0b8" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 38427 } health_report { overall_health: UNKNOWN } } }
I20260812 09:45:14.363839 16520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:14.359773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.2:35803 (local address 127.0.0.1:59236)
0812 09:45:14.360033 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:14.360052 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:14.360170 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:14.360559 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:14.360562 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:14.360575 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 09:45:14.360691 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:14.360697 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:14.361481 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:14.361485 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:14.362112 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:14.362121 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:14.362616 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:14.362620 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:14.362718 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:14.362770 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 09:45:14.363503 (+ 733us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 09:45:14.363522 (+ 19us) client_negotiation.cc:769] Sending connection context
0812 09:45:14.363597 (+ 75us) client_negotiation.cc:240] Negotiation successful
0812 09:45:14.363725 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138}
I20260812 09:45:14.363852 16533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:14.359888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59236 (local address 127.13.99.2:35803)
0812 09:45:14.360134 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:14.360139 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:14.360154 (+ 15us) server_negotiation.cc:407] Connection header received
0812 09:45:14.360338 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:14.360341 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:14.360395 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:14.360557 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 09:45:14.360814 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:14.361364 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:14.362268 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:14.362495 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:14.362616 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:14.362932 (+ 316us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 09:45:14.363401 (+ 469us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 09:45:14.363542 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:14.363607 (+ 65us) server_negotiation.cc:299] Negotiation successful
0812 09:45:14.363685 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":78,"threads_started":1}
I20260812 09:45:14.367146 16335 raft_consensus.cc:1261] T 048b56eb17fc421c845e82b2e7546f61 P 54f7fe0d3da94da18132fb2d7b0ed0b8 [term 1 FOLLOWER]: Refusing update from remote peer 0631d48e67d34bcb84ab4ecb65163865: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 09:45:14.367342 15864 raft_consensus.cc:1261] T 048b56eb17fc421c845e82b2e7546f61 P 981793cd97244da8b9e2b369c1892d32 [term 1 FOLLOWER]: Refusing update from remote peer 0631d48e67d34bcb84ab4ecb65163865: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 09:45:14.367669 16466 consensus_queue.cc:1035] T 048b56eb17fc421c845e82b2e7546f61 P 0631d48e67d34bcb84ab4ecb65163865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54f7fe0d3da94da18132fb2d7b0ed0b8" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 38427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 09:45:14.368068 16466 consensus_queue.cc:1035] T 048b56eb17fc421c845e82b2e7546f61 P 0631d48e67d34bcb84ab4ecb65163865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "981793cd97244da8b9e2b369c1892d32" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 33687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 09:45:14.369930 16491 tablet_replica.cc:439] T 048b56eb17fc421c845e82b2e7546f61 P 981793cd97244da8b9e2b369c1892d32: TxnStatusTablet state changed. Reason: New leader 0631d48e67d34bcb84ab4ecb65163865. Latest consensus state: current_term: 1 leader_uuid: "0631d48e67d34bcb84ab4ecb65163865" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "981793cd97244da8b9e2b369c1892d32" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 33687 } } peers { permanent_uuid: "0631d48e67d34bcb84ab4ecb65163865" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35803 } } peers { permanent_uuid: "54f7fe0d3da94da18132fb2d7b0ed0b8" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 38427 } } }
I20260812 09:45:14.370051 16491 tablet_replica.cc:442] T 048b56eb17fc421c845e82b2e7546f61 P 981793cd97244da8b9e2b369c1892d32: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 09:45:14.371207 16469 tablet_replica.cc:439] T 048b56eb17fc421c845e82b2e7546f61 P 54f7fe0d3da94da18132fb2d7b0ed0b8: TxnStatusTablet state changed. Reason: New leader 0631d48e67d34bcb84ab4ecb65163865. Latest consensus state: current_term: 1 leader_uuid: "0631d48e67d34bcb84ab4ecb65163865" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "981793cd97244da8b9e2b369c1892d32" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 33687 } } peers { permanent_uuid: "0631d48e67d34bcb84ab4ecb65163865" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35803 } } peers { permanent_uuid: "54f7fe0d3da94da18132fb2d7b0ed0b8" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 38427 } } }
I20260812 09:45:14.371299 16469 tablet_replica.cc:442] T 048b56eb17fc421c845e82b2e7546f61 P 54f7fe0d3da94da18132fb2d7b0ed0b8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 09:45:14.372857 16466 tablet_replica.cc:439] T 048b56eb17fc421c845e82b2e7546f61 P 0631d48e67d34bcb84ab4ecb65163865: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0631d48e67d34bcb84ab4ecb65163865" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "981793cd97244da8b9e2b369c1892d32" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 33687 } } peers { permanent_uuid: "0631d48e67d34bcb84ab4ecb65163865" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35803 } } peers { permanent_uuid: "54f7fe0d3da94da18132fb2d7b0ed0b8" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 38427 } } }
I20260812 09:45:14.373013 16466 tablet_replica.cc:442] T 048b56eb17fc421c845e82b2e7546f61 P 0631d48e67d34bcb84ab4ecb65163865: This TxnStatusTablet replica's current role is: LEADER
I20260812 09:45:14.373862 16469 tablet_replica.cc:439] T 048b56eb17fc421c845e82b2e7546f61 P 54f7fe0d3da94da18132fb2d7b0ed0b8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0631d48e67d34bcb84ab4ecb65163865" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "981793cd97244da8b9e2b369c1892d32" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 33687 } } peers { permanent_uuid: "0631d48e67d34bcb84ab4ecb65163865" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35803 } } peers { permanent_uuid: "54f7fe0d3da94da18132fb2d7b0ed0b8" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 38427 } } }
I20260812 09:45:14.373976 16469 tablet_replica.cc:442] T 048b56eb17fc421c845e82b2e7546f61 P 54f7fe0d3da94da18132fb2d7b0ed0b8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 09:45:14.374851 16466 tablet_replica.cc:439] T 048b56eb17fc421c845e82b2e7546f61 P 0631d48e67d34bcb84ab4ecb65163865: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0631d48e67d34bcb84ab4ecb65163865" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "981793cd97244da8b9e2b369c1892d32" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 33687 } } peers { permanent_uuid: "0631d48e67d34bcb84ab4ecb65163865" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35803 } } peers { permanent_uuid: "54f7fe0d3da94da18132fb2d7b0ed0b8" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 38427 } } }
I20260812 09:45:14.374951 16466 tablet_replica.cc:442] T 048b56eb17fc421c845e82b2e7546f61 P 0631d48e67d34bcb84ab4ecb65163865: This TxnStatusTablet replica's current role is: LEADER
I20260812 09:45:14.376016 16491 tablet_replica.cc:439] T 048b56eb17fc421c845e82b2e7546f61 P 981793cd97244da8b9e2b369c1892d32: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0631d48e67d34bcb84ab4ecb65163865" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "981793cd97244da8b9e2b369c1892d32" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 33687 } } peers { permanent_uuid: "0631d48e67d34bcb84ab4ecb65163865" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35803 } } peers { permanent_uuid: "54f7fe0d3da94da18132fb2d7b0ed0b8" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 38427 } } }
I20260812 09:45:14.376133 16491 tablet_replica.cc:442] T 048b56eb17fc421c845e82b2e7546f61 P 981793cd97244da8b9e2b369c1892d32: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 09:45:14.389799 16533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:14.385738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59238 (local address 127.13.99.2:35803)
0812 09:45:14.385889 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:14.385894 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:14.385997 (+ 103us) server_negotiation.cc:407] Connection header received
0812 09:45:14.386105 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:14.386108 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:14.386222 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:14.386314 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 09:45:14.386539 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:14.387322 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:14.388199 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:14.388566 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:14.388674 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:14.389080 (+ 406us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 09:45:14.389172 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 09:45:14.389260 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:14.389599 (+ 339us) server_negotiation.cc:299] Negotiation successful
0812 09:45:14.389657 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260812 09:45:14.401393 16533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:14.397289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.2:40679 (local address 127.13.99.2:35803)
0812 09:45:14.397419 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:14.397423 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:14.398047 (+ 624us) server_negotiation.cc:407] Connection header received
0812 09:45:14.398189 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:14.398192 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:14.398243 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:14.398323 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 09:45:14.398553 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:14.399328 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:14.400209 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:14.400566 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:14.400661 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:14.400949 (+ 288us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 09:45:14.401035 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 09:45:14.401107 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:14.401207 (+ 100us) server_negotiation.cc:299] Negotiation successful
0812 09:45:14.401255 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260812 09:45:14.401473 16559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:14.397647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.2:35803 (local address 127.13.99.2:40679)
0812 09:45:14.397944 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:14.397957 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:14.398081 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:14.398339 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:14.398341 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:14.398348 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 09:45:14.398452 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:14.398456 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:14.399442 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:14.399445 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:14.400094 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:14.400103 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:14.400690 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:14.400693 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:14.400766 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:14.400814 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 09:45:14.401122 (+ 308us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 09:45:14.401127 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 09:45:14.401180 (+ 53us) client_negotiation.cc:240] Negotiation successful
0812 09:45:14.401241 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":115,"threads_started":1}
I20260812 09:45:14.408354 15864 raft_consensus.cc:1261] T f1eac8933c884a32b98ef4c6627d4c4a P 981793cd97244da8b9e2b369c1892d32 [term 1 FOLLOWER]: Refusing update from remote peer 0631d48e67d34bcb84ab4ecb65163865: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 09:45:14.408349 16335 raft_consensus.cc:1261] T f1eac8933c884a32b98ef4c6627d4c4a P 54f7fe0d3da94da18132fb2d7b0ed0b8 [term 1 FOLLOWER]: Refusing update from remote peer 0631d48e67d34bcb84ab4ecb65163865: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 09:45:14.408829 16511 consensus_queue.cc:1035] T f1eac8933c884a32b98ef4c6627d4c4a P 0631d48e67d34bcb84ab4ecb65163865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54f7fe0d3da94da18132fb2d7b0ed0b8" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 38427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 09:45:14.409017 16466 consensus_queue.cc:1035] T f1eac8933c884a32b98ef4c6627d4c4a P 0631d48e67d34bcb84ab4ecb65163865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "981793cd97244da8b9e2b369c1892d32" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 33687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 09:45:14.454082 15587 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46588:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 09:45:14.454232 15587 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46588:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 09:45:14.456313 15587 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3eac478794a6474684ce396fa014045f: Sending DeleteTablet for 3 replicas of tablet f1eac8933c884a32b98ef4c6627d4c4a
I20260812 09:45:14.456893 15831 tablet_service.cc:1514] Processing DeleteTablet for tablet f1eac8933c884a32b98ef4c6627d4c4a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 09:45:14 UTC) from {username='kudu', principal='kudu/127.13.99.62@KRBTEST.COM'} at 127.0.0.1:51680
I20260812 09:45:14.456954 16048 tablet_service.cc:1514] Processing DeleteTablet for tablet f1eac8933c884a32b98ef4c6627d4c4a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 09:45:14 UTC) from {username='kudu', principal='kudu/127.13.99.62@KRBTEST.COM'} at 127.0.0.1:59222
I20260812 09:45:14.457113 16300 tablet_service.cc:1514] Processing DeleteTablet for tablet f1eac8933c884a32b98ef4c6627d4c4a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 09:45:14 UTC) from {username='kudu', principal='kudu/127.13.99.62@KRBTEST.COM'} at 127.0.0.1:57466
I20260812 09:45:14.457458 16567 tablet_replica.cc:330] T f1eac8933c884a32b98ef4c6627d4c4a P 0631d48e67d34bcb84ab4ecb65163865: stopping tablet replica
I20260812 09:45:14.457602 16567 raft_consensus.cc:2229] T f1eac8933c884a32b98ef4c6627d4c4a P 0631d48e67d34bcb84ab4ecb65163865 [term 1 LEADER]: Raft consensus shutting down.
I20260812 09:45:14.457913 16567 raft_consensus.cc:2258] T f1eac8933c884a32b98ef4c6627d4c4a P 0631d48e67d34bcb84ab4ecb65163865 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 09:45:14.458173 16566 tablet_replica.cc:330] T f1eac8933c884a32b98ef4c6627d4c4a P 981793cd97244da8b9e2b369c1892d32: stopping tablet replica
I20260812 09:45:14.458340 16566 raft_consensus.cc:2229] T f1eac8933c884a32b98ef4c6627d4c4a P 981793cd97244da8b9e2b369c1892d32 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 09:45:14.458487 16566 raft_consensus.cc:2258] T f1eac8933c884a32b98ef4c6627d4c4a P 981793cd97244da8b9e2b369c1892d32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 09:45:14.459152 16566 ts_tablet_manager.cc:1905] T f1eac8933c884a32b98ef4c6627d4c4a P 981793cd97244da8b9e2b369c1892d32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 09:45:14.459152 16567 ts_tablet_manager.cc:1905] T f1eac8933c884a32b98ef4c6627d4c4a P 0631d48e67d34bcb84ab4ecb65163865: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 09:45:14.462100 16566 ts_tablet_manager.cc:1918] T f1eac8933c884a32b98ef4c6627d4c4a P 981793cd97244da8b9e2b369c1892d32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 09:45:14.462100 16567 ts_tablet_manager.cc:1918] T f1eac8933c884a32b98ef4c6627d4c4a P 0631d48e67d34bcb84ab4ecb65163865: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 09:45:14.462188 16566 log.cc:1196] T f1eac8933c884a32b98ef4c6627d4c4a P 981793cd97244da8b9e2b369c1892d32: Deleting WAL directory at /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/minicluster-data/ts-0/wal/wals/f1eac8933c884a32b98ef4c6627d4c4a
I20260812 09:45:14.462188 16567 log.cc:1196] T f1eac8933c884a32b98ef4c6627d4c4a P 0631d48e67d34bcb84ab4ecb65163865: Deleting WAL directory at /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/minicluster-data/ts-1/wal/wals/f1eac8933c884a32b98ef4c6627d4c4a
I20260812 09:45:14.462473 16566 ts_tablet_manager.cc:1939] T f1eac8933c884a32b98ef4c6627d4c4a P 981793cd97244da8b9e2b369c1892d32: Deleting consensus metadata
I20260812 09:45:14.462473 16567 ts_tablet_manager.cc:1939] T f1eac8933c884a32b98ef4c6627d4c4a P 0631d48e67d34bcb84ab4ecb65163865: Deleting consensus metadata
I20260812 09:45:14.463191 16570 tablet_replica.cc:330] T f1eac8933c884a32b98ef4c6627d4c4a P 54f7fe0d3da94da18132fb2d7b0ed0b8: stopping tablet replica
I20260812 09:45:14.463271 15569 catalog_manager.cc:4841] TS 0631d48e67d34bcb84ab4ecb65163865 (127.13.99.2:35803): tablet f1eac8933c884a32b98ef4c6627d4c4a (table test-table [id=3d0e63f262634573b99c60d1a195ba79]) successfully deleted
I20260812 09:45:14.463338 16570 raft_consensus.cc:2229] T f1eac8933c884a32b98ef4c6627d4c4a P 54f7fe0d3da94da18132fb2d7b0ed0b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 09:45:14.463268 15567 catalog_manager.cc:4841] TS 981793cd97244da8b9e2b369c1892d32 (127.13.99.1:33687): tablet f1eac8933c884a32b98ef4c6627d4c4a (table test-table [id=3d0e63f262634573b99c60d1a195ba79]) successfully deleted
I20260812 09:45:14.463487 16570 raft_consensus.cc:2258] T f1eac8933c884a32b98ef4c6627d4c4a P 54f7fe0d3da94da18132fb2d7b0ed0b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 09:45:14.464315 16570 ts_tablet_manager.cc:1905] T f1eac8933c884a32b98ef4c6627d4c4a P 54f7fe0d3da94da18132fb2d7b0ed0b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 09:45:14.467140 16570 ts_tablet_manager.cc:1918] T f1eac8933c884a32b98ef4c6627d4c4a P 54f7fe0d3da94da18132fb2d7b0ed0b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 09:45:14.467252 16570 log.cc:1196] T f1eac8933c884a32b98ef4c6627d4c4a P 54f7fe0d3da94da18132fb2d7b0ed0b8: Deleting WAL directory at /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786527905902985-13708-0/minicluster-data/ts-2/wal/wals/f1eac8933c884a32b98ef4c6627d4c4a
I20260812 09:45:14.467509 16570 ts_tablet_manager.cc:1939] T f1eac8933c884a32b98ef4c6627d4c4a P 54f7fe0d3da94da18132fb2d7b0ed0b8: Deleting consensus metadata
I20260812 09:45:14.468294 15568 catalog_manager.cc:4841] TS 54f7fe0d3da94da18132fb2d7b0ed0b8 (127.13.99.3:38427): tablet f1eac8933c884a32b98ef4c6627d4c4a (table test-table [id=3d0e63f262634573b99c60d1a195ba79]) successfully deleted
Aug 12 09:45:14 dist-test-slave-jq4t krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527913, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.13.99.1@KRBTEST.COM
I20260812 09:45:14.475543 16451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:14.458860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51712 (local address 127.13.99.1:33687)
0812 09:45:14.459043 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:14.459048 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:14.463902 (+ 4854us) server_negotiation.cc:407] Connection header received
0812 09:45:14.464030 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:14.464033 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:14.464086 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:14.464186 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:14.465133 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:14.465949 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:14.467157 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:14.467536 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:14.467649 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:14.471665 (+ 4016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:14.471679 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:14.471696 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:14.471725 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:14.473656 (+ 1931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:14.474082 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:14.474090 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:14.474094 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:14.474191 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:14.474525 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:14.474529 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:14.474531 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:14.474859 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:14.475090 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:14.475245 (+ 155us) server_negotiation.cc:299] Negotiation successful
0812 09:45:14.475378 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
W20260812 09:45:14.476095 15931 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51712
I20260812 09:45:14.486039 16521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:14.477942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46608 (local address 127.13.99.62:32955)
0812 09:45:14.478067 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:14.478070 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:14.478276 (+ 206us) server_negotiation.cc:407] Connection header received
0812 09:45:14.478348 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:14.478351 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:14.478410 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:14.478529 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:14.479116 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:14.479942 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:14.480822 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:14.481233 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:14.481347 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:14.482650 (+ 1303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:14.482654 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:14.482657 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:14.482684 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:14.484271 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:14.484764 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:14.484768 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:14.484770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:14.484867 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:14.485156 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:14.485159 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:14.485161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:14.485383 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:14.485526 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:14.485730 (+ 204us) server_negotiation.cc:299] Negotiation successful
0812 09:45:14.485867 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260812 09:45:14.486435 15587 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46608
I20260812 09:45:14.487838 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 15706
I20260812 09:45:14.509173 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 15959
I20260812 09:45:14.523595 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 16192
I20260812 09:45:14.540040 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 15544
2026-08-12T09:45:14Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3345 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 09:45:14 dist-test-slave-jq4t krb5kdc[16579](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 09:45:14 dist-test-slave-jq4t krb5kdc[16579](info): set up 2 sockets
Aug 12 09:45:14 dist-test-slave-jq4t krb5kdc[16579](info): commencing operation
krb5kdc: starting...
W20260812 09:45:15.823555 13708 mini_kdc.cc:121] Time spent starting KDC: real 1.253s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/krb5kdc/test-user.keytab.
Aug 12 09:45:15 dist-test-slave-jq4t krb5kdc[16579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527915, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T09:45:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T09:45:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.99.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.99.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.99.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab.
Entry for principal kudu/127.13.99.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab.
I20260812 09:45:15.955506 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.62:37397
--webserver_interface=127.13.99.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab
--principal=kudu/127.13.99.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:35367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.99.62:37397
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:45:16.068845 16599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:45:16.069126 16599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:45:16.069199 16599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:45:16.073161 16599 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 09:45:16.073246 16599 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260812 09:45:16.073277 16599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:45:16.073321 16599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:45:16.073359 16599 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 09:45:16.073395 16599 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 09:45:16.078433 16599 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:35367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.99.62:37397
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab
--principal=kudu/127.13.99.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.62:37397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.99.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16599
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:45:16.079466 16599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:45:16.080534 16599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:45:16.088976 16611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:45:16.089167 16599 server_base.cc:1034] running on GCE node
W20260812 09:45:16.088999 16607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:16.089323 16609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:45:16.089915 16599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:45:16.090960 16599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:45:16.092152 16599 hybrid_clock.cc:648] HybridClock initialized: now 1786527916092131 us; error 31 us; skew 500 ppm
Aug 12 09:45:16 dist-test-slave-jq4t krb5kdc[16579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527916, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:45:16.094967 16599 init.cc:377] Logged in from keytab as kudu/127.13.99.62@KRBTEST.COM (short username kudu)
I20260812 09:45:16.095781 16599 webserver.cc:460] Webserver started at http://127.13.99.62:35119/ using document root <none> and password file <none>
I20260812 09:45:16.096287 16599 fs_manager.cc:359] Metadata directory not provided
I20260812 09:45:16.096344 16599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:45:16.096585 16599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:45:16.098613 16599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/minicluster-data/master-0/data/instance:
uuid: "77dec44a5cf34f3a9c0e1dfc5ff57cd4"
format_stamp: "Formatted at 2026-08-12 09:45:16 on dist-test-slave-jq4t"
server_key: "5f8156c94989ce1472852741ff001baa"
server_key_iv: "df4b599fb28cce15125da37325432eea"
server_key_version: "encryptionkey@0"
I20260812 09:45:16.099152 16599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/minicluster-data/master-0/wal/instance:
uuid: "77dec44a5cf34f3a9c0e1dfc5ff57cd4"
format_stamp: "Formatted at 2026-08-12 09:45:16 on dist-test-slave-jq4t"
server_key: "5f8156c94989ce1472852741ff001baa"
server_key_iv: "df4b599fb28cce15125da37325432eea"
server_key_version: "encryptionkey@0"
I20260812 09:45:16.102893 16599 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 09:45:16.105252 16620 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:16.106491 16599 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 09:45:16.106594 16599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/minicluster-data/master-0/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/minicluster-data/master-0/wal
uuid: "77dec44a5cf34f3a9c0e1dfc5ff57cd4"
format_stamp: "Formatted at 2026-08-12 09:45:16 on dist-test-slave-jq4t"
server_key: "5f8156c94989ce1472852741ff001baa"
server_key_iv: "df4b599fb28cce15125da37325432eea"
server_key_version: "encryptionkey@0"
I20260812 09:45:16.106678 16599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:45:16.116899 16599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:45:16.119529 16599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:45:16.119679 16599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:45:16.127661 16599 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.62:37397
I20260812 09:45:16.127759 16736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.62:37397 every 8 connection(s)
I20260812 09:45:16.128747 16599 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
I20260812 09:45:16.131803 16737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:45:16.131954 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 16599
I20260812 09:45:16.132086 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/minicluster-data/master-0/wal/instance
I20260812 09:45:16.132390 13708 external_mini_cluster.cc:1428] Setting key 75ab7ce363a3e43e58af0d6bd52a3180
I20260812 09:45:16.139315 16737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77dec44a5cf34f3a9c0e1dfc5ff57cd4: Bootstrap starting.
I20260812 09:45:16.142350 16737 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 77dec44a5cf34f3a9c0e1dfc5ff57cd4: Neither blocks nor log segments found. Creating new log.
I20260812 09:45:16.143153 16737 log.cc:824] T 00000000000000000000000000000000 P 77dec44a5cf34f3a9c0e1dfc5ff57cd4: Log is configured to *not* fsync() on all Append() calls
I20260812 09:45:16.146304 16737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77dec44a5cf34f3a9c0e1dfc5ff57cd4: No bootstrap required, opened a new log
Aug 12 09:45:16 dist-test-slave-jq4t krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527915, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
I20260812 09:45:16.150856 16737 raft_consensus.cc:348] T 00000000000000000000000000000000 P 77dec44a5cf34f3a9c0e1dfc5ff57cd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77dec44a5cf34f3a9c0e1dfc5ff57cd4" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 37397 } }
I20260812 09:45:16.151139 16737 raft_consensus.cc:374] T 00000000000000000000000000000000 P 77dec44a5cf34f3a9c0e1dfc5ff57cd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:45:16.151225 16737 raft_consensus.cc:729] T 00000000000000000000000000000000 P 77dec44a5cf34f3a9c0e1dfc5ff57cd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77dec44a5cf34f3a9c0e1dfc5ff57cd4, State: Initialized, Role: FOLLOWER
I20260812 09:45:16.151741 16737 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77dec44a5cf34f3a9c0e1dfc5ff57cd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77dec44a5cf34f3a9c0e1dfc5ff57cd4" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 37397 } }
I20260812 09:45:16.151896 16737 raft_consensus.cc:388] T 00000000000000000000000000000000 P 77dec44a5cf34f3a9c0e1dfc5ff57cd4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 09:45:16.151984 16737 raft_consensus.cc:482] T 00000000000000000000000000000000 P 77dec44a5cf34f3a9c0e1dfc5ff57cd4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 09:45:16.152098 16737 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 77dec44a5cf34f3a9c0e1dfc5ff57cd4 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:45:16.153332 16737 raft_consensus.cc:504] T 00000000000000000000000000000000 P 77dec44a5cf34f3a9c0e1dfc5ff57cd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77dec44a5cf34f3a9c0e1dfc5ff57cd4" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 37397 } }
I20260812 09:45:16.153700 16737 leader_election.cc:302] T 00000000000000000000000000000000 P 77dec44a5cf34f3a9c0e1dfc5ff57cd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 77dec44a5cf34f3a9c0e1dfc5ff57cd4; no voters:
I20260812 09:45:16.154050 16737 leader_election.cc:288] T 00000000000000000000000000000000 P 77dec44a5cf34f3a9c0e1dfc5ff57cd4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 09:45:16.154168 16750 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 77dec44a5cf34f3a9c0e1dfc5ff57cd4 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 09:45:16.154397 16750 raft_consensus.cc:686] T 00000000000000000000000000000000 P 77dec44a5cf34f3a9c0e1dfc5ff57cd4 [term 1 LEADER]: Becoming Leader. State: Replica: 77dec44a5cf34f3a9c0e1dfc5ff57cd4, State: Running, Role: LEADER
I20260812 09:45:16.154708 16750 consensus_queue.cc:237] T 00000000000000000000000000000000 P 77dec44a5cf34f3a9c0e1dfc5ff57cd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77dec44a5cf34f3a9c0e1dfc5ff57cd4" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 37397 } }
I20260812 09:45:16.154877 16743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:16.133810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43096 (local address 127.13.99.62:37397)
0812 09:45:16.134278 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:16.134305 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:16.134368 (+ 63us) server_negotiation.cc:407] Connection header received
0812 09:45:16.136145 (+ 1777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:16.136172 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:16.136654 (+ 482us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:16.137030 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:16.138282 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:16.139781 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:16.141112 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:16.141810 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:16.141963 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:16.149525 (+ 7562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:16.149531 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:16.149546 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:16.149576 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:16.151489 (+ 1913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:16.152012 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:16.152018 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:16.152026 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:16.152188 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:16.152508 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:16.152511 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:16.152513 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:16.153057 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:16.153290 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:16.153480 (+ 190us) server_negotiation.cc:299] Negotiation successful
0812 09:45:16.153798 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":163,"threads_started":1}
I20260812 09:45:16.155385 16737 sys_catalog.cc:564] T 00000000000000000000000000000000 P 77dec44a5cf34f3a9c0e1dfc5ff57cd4 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 09:45:16.156333 16751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77dec44a5cf34f3a9c0e1dfc5ff57cd4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "77dec44a5cf34f3a9c0e1dfc5ff57cd4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77dec44a5cf34f3a9c0e1dfc5ff57cd4" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 37397 } } }
I20260812 09:45:16.156466 16751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77dec44a5cf34f3a9c0e1dfc5ff57cd4 [sys.catalog]: This master's current role is: LEADER
I20260812 09:45:16.156816 16752 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77dec44a5cf34f3a9c0e1dfc5ff57cd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 77dec44a5cf34f3a9c0e1dfc5ff57cd4. Latest consensus state: current_term: 1 leader_uuid: "77dec44a5cf34f3a9c0e1dfc5ff57cd4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77dec44a5cf34f3a9c0e1dfc5ff57cd4" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 37397 } } }
I20260812 09:45:16.156911 16752 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77dec44a5cf34f3a9c0e1dfc5ff57cd4 [sys.catalog]: This master's current role is: LEADER
I20260812 09:45:16.157780 16762 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 09:45:16.160995 16762 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 09:45:16.166532 16762 catalog_manager.cc:1347] Generated new cluster ID: d819aa1ddd4c451c83d7d6ef0b2f4e93
I20260812 09:45:16.166610 16762 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 09:45:16.173183 16762 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 09:45:16.174247 16762 catalog_manager.cc:1504] Loading token signing keys...
I20260812 09:45:16.182847 16762 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 77dec44a5cf34f3a9c0e1dfc5ff57cd4: Generated new TSK 0
I20260812 09:45:16.183553 16762 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.99.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.99.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.99.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.1.keytab.
Entry for principal kudu/127.13.99.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.1.keytab.
I20260812 09:45:16.223373 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.99.1:0
--local_ip_for_outbound_sockets=127.13.99.1
--webserver_interface=127.13.99.1
--webserver_port=0
--tserver_master_addrs=127.13.99.62:37397
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.1.keytab
--principal=kudu/127.13.99.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:35367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:45:16.339157 16780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:45:16.339481 16780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:45:16.339535 16780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:45:16.343401 16780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:45:16.343447 16780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:45:16.343489 16780 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260812 09:45:16.343504 16780 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 09:45:16.343534 16780 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.99.1
I20260812 09:45:16.348534 16780 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:35367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.1.keytab
--principal=kudu/127.13.99.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.99.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.99.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.99.62:37397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16780
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.99.1
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:45:16.349503 16780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:45:16.350553 16780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:45:16.360817 16795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:16.360817 16789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:16.360857 16790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:45:16.361016 16780 server_base.cc:1034] running on GCE node
I20260812 09:45:16.361491 16780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:45:16.362174 16780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:45:16.363377 16780 hybrid_clock.cc:648] HybridClock initialized: now 1786527916363360 us; error 28 us; skew 500 ppm
Aug 12 09:45:16 dist-test-slave-jq4t krb5kdc[16579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527916, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:45:16.366376 16780 init.cc:377] Logged in from keytab as kudu/127.13.99.1@KRBTEST.COM (short username kudu)
I20260812 09:45:16.367298 16780 webserver.cc:460] Webserver started at http://127.13.99.1:44725/ using document root <none> and password file <none>
I20260812 09:45:16.367897 16780 fs_manager.cc:359] Metadata directory not provided
I20260812 09:45:16.368003 16780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:45:16.368252 16780 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:45:16.370481 16780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/minicluster-data/ts-0/data/instance:
uuid: "9a386c33a01a49e4adf0dbcd90cf9223"
format_stamp: "Formatted at 2026-08-12 09:45:16 on dist-test-slave-jq4t"
server_key: "fd46cb73e7007a3ff28bdce1abe79f80"
server_key_iv: "86662a09a4368d4cb50205b3060b5dff"
server_key_version: "encryptionkey@0"
I20260812 09:45:16.371111 16780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/minicluster-data/ts-0/wal/instance:
uuid: "9a386c33a01a49e4adf0dbcd90cf9223"
format_stamp: "Formatted at 2026-08-12 09:45:16 on dist-test-slave-jq4t"
server_key: "fd46cb73e7007a3ff28bdce1abe79f80"
server_key_iv: "86662a09a4368d4cb50205b3060b5dff"
server_key_version: "encryptionkey@0"
I20260812 09:45:16.375419 16780 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260812 09:45:16.378034 16803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:16.379395 16780 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260812 09:45:16.379524 16780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/minicluster-data/ts-0/wal
uuid: "9a386c33a01a49e4adf0dbcd90cf9223"
format_stamp: "Formatted at 2026-08-12 09:45:16 on dist-test-slave-jq4t"
server_key: "fd46cb73e7007a3ff28bdce1abe79f80"
server_key_iv: "86662a09a4368d4cb50205b3060b5dff"
server_key_version: "encryptionkey@0"
I20260812 09:45:16.379653 16780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:45:16.406638 16780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:45:16.410419 16780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:45:16.410689 16780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:45:16.412267 16780 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:45:16.412345 16780 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:16.412423 16780 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:45:16.412469 16780 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 09:45:16 dist-test-slave-jq4t krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527916, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.1@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
I20260812 09:45:16.424088 16780 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.1:42111
I20260812 09:45:16.424206 17011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.1:42111 every 8 connection(s)
I20260812 09:45:16.425104 16780 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/minicluster-data/ts-0/data/info.pb
I20260812 09:45:16.427031 16743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:16.413298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.1:56417 (local address 127.13.99.62:37397)
0812 09:45:16.413455 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:16.413460 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:16.414428 (+ 968us) server_negotiation.cc:407] Connection header received
0812 09:45:16.415611 (+ 1183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:16.415618 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:16.415689 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:16.415796 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:16.417447 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:16.417998 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:16.419500 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:16.419805 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:16.419936 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:16.423253 (+ 3317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:16.423256 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:16.423259 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:16.423285 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:16.424979 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:16.425528 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:16.425532 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:16.425534 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:16.425641 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:16.426044 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:16.426047 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:16.426048 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:16.426237 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:16.426325 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:16.426772 (+ 447us) server_negotiation.cc:299] Negotiation successful
0812 09:45:16.426880 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260812 09:45:16.427810 16821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:16.413743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:37397 (local address 127.13.99.1:56417)
0812 09:45:16.414239 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:16.414291 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:16.415324 (+ 1033us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:16.416022 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:16.416033 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:16.416567 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:45:16.417282 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:16.417300 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:16.418138 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:16.418142 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:16.419350 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:16.419361 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:16.419994 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:16.419999 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:16.420187 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:16.420858 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:45:16.420878 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:45:16.423070 (+ 2192us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:45:16.425107 (+ 2037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:16.425113 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:16.425127 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:16.425413 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:16.425753 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:16.425756 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:16.425758 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:16.425935 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:16.426341 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:45:16.426345 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:45:16.426566 (+ 221us) client_negotiation.cc:769] Sending connection context
0812 09:45:16.426716 (+ 150us) client_negotiation.cc:240] Negotiation successful
0812 09:45:16.426957 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":140,"threads_started":1}
I20260812 09:45:16.429210 17012 heartbeater.cc:347] Connected to a master server at 127.13.99.62:37397
I20260812 09:45:16.429365 17012 heartbeater.cc:464] Registering TS with master...
I20260812 09:45:16.429859 17012 heartbeater.cc:511] Master 127.13.99.62:37397 requested a full tablet report, sending...
I20260812 09:45:16.431042 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 16780
I20260812 09:45:16.431186 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/minicluster-data/ts-0/wal/instance
I20260812 09:45:16.431492 13708 external_mini_cluster.cc:1428] Setting key d76ce159cd2a5015d8a1f6cb81cdb5aa
I20260812 09:45:16.431525 16643 ts_manager.cc:194] Registered new tserver with Master: 9a386c33a01a49e4adf0dbcd90cf9223 (127.13.99.1:42111)
I20260812 09:45:16.432755 16643 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.99.1@KRBTEST.COM'} at 127.13.99.1:56417
I20260812 09:45:16.440570 16743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:16.433528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.1:57847 (local address 127.13.99.62:37397)
0812 09:45:16.433641 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:16.433644 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:16.433870 (+ 226us) server_negotiation.cc:407] Connection header received
0812 09:45:16.433983 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:16.433985 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:16.434020 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:16.434082 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:16.434630 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:16.435120 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:16.435942 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:16.436168 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:16.436273 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:16.437415 (+ 1142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:16.437419 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:16.437421 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:16.437444 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:16.438941 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:16.439361 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:16.439365 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:16.439367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:16.439451 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:16.439734 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:16.439738 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:16.439740 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:16.439918 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:16.440026 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:16.440288 (+ 262us) server_negotiation.cc:299] Negotiation successful
0812 09:45:16.440400 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260812 09:45:16.440590 17024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:16.433493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:37397 (local address 127.13.99.1:57847)
0812 09:45:16.433776 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:16.433794 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:16.433901 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:16.434119 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:16.434125 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:16.434355 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:45:16.434509 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:16.434519 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:16.435253 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:16.435256 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:16.435818 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:16.435826 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:16.436289 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:16.436292 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:16.436393 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:16.436941 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:45:16.436961 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:45:16.437268 (+ 307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:45:16.439053 (+ 1785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:16.439058 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:16.439060 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:16.439259 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:16.439546 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:16.439550 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:16.439552 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:16.439640 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:16.440017 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:45:16.440020 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:45:16.440120 (+ 100us) client_negotiation.cc:769] Sending connection context
0812 09:45:16.440232 (+ 112us) client_negotiation.cc:240] Negotiation successful
0812 09:45:16.440373 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":94,"threads_started":1}
WARNING: no policy specified for kudu/127.13.99.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.99.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.99.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.2.keytab.
Entry for principal kudu/127.13.99.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.2.keytab.
I20260812 09:45:16.463068 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.99.2:0
--local_ip_for_outbound_sockets=127.13.99.2
--webserver_interface=127.13.99.2
--webserver_port=0
--tserver_master_addrs=127.13.99.62:37397
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.2.keytab
--principal=kudu/127.13.99.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:35367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:45:16.579754 17031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:45:16.580005 17031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:45:16.580072 17031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:45:16.584183 17031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:45:16.584237 17031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:45:16.584277 17031 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260812 09:45:16.584291 17031 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 09:45:16.584322 17031 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.99.2
I20260812 09:45:16.589545 17031 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:35367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.2.keytab
--principal=kudu/127.13.99.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.99.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.99.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.99.62:37397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17031
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.99.2
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:45:16.590531 17031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:45:16.591540 17031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:45:16.601183 17040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:16.601154 17043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:16.601169 17041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:45:16.601436 17031 server_base.cc:1034] running on GCE node
I20260812 09:45:16.601995 17031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:45:16.602639 17031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:45:16.603801 17031 hybrid_clock.cc:648] HybridClock initialized: now 1786527916603794 us; error 38 us; skew 500 ppm
Aug 12 09:45:16 dist-test-slave-jq4t krb5kdc[16579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527916, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:45:16.606676 17031 init.cc:377] Logged in from keytab as kudu/127.13.99.2@KRBTEST.COM (short username kudu)
I20260812 09:45:16.607654 17031 webserver.cc:460] Webserver started at http://127.13.99.2:36391/ using document root <none> and password file <none>
I20260812 09:45:16.608253 17031 fs_manager.cc:359] Metadata directory not provided
I20260812 09:45:16.608353 17031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:45:16.608654 17031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:45:16.610888 17031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/minicluster-data/ts-1/data/instance:
uuid: "ca5cccf002874a15b99a3ec1fdea4c3f"
format_stamp: "Formatted at 2026-08-12 09:45:16 on dist-test-slave-jq4t"
server_key: "ff56b10309c9881321ebeb9561471821"
server_key_iv: "e98f347911bf2b473bc2bd601fa9489c"
server_key_version: "encryptionkey@0"
I20260812 09:45:16.611471 17031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/minicluster-data/ts-1/wal/instance:
uuid: "ca5cccf002874a15b99a3ec1fdea4c3f"
format_stamp: "Formatted at 2026-08-12 09:45:16 on dist-test-slave-jq4t"
server_key: "ff56b10309c9881321ebeb9561471821"
server_key_iv: "e98f347911bf2b473bc2bd601fa9489c"
server_key_version: "encryptionkey@0"
I20260812 09:45:16.615373 17031 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260812 09:45:16.618049 17053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:16.619310 17031 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260812 09:45:16.619464 17031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/minicluster-data/ts-1/wal
uuid: "ca5cccf002874a15b99a3ec1fdea4c3f"
format_stamp: "Formatted at 2026-08-12 09:45:16 on dist-test-slave-jq4t"
server_key: "ff56b10309c9881321ebeb9561471821"
server_key_iv: "e98f347911bf2b473bc2bd601fa9489c"
server_key_version: "encryptionkey@0"
I20260812 09:45:16.619607 17031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:45:16.631356 17031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:45:16.634547 17031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:45:16.634763 17031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:45:16.636344 17031 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:45:16.636420 17031 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:16.636484 17031 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:45:16.636602 17031 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
Aug 12 09:45:16 dist-test-slave-jq4t krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527916, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.2@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
I20260812 09:45:16.647822 17031 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.2:35267
I20260812 09:45:16.648988 17031 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/minicluster-data/ts-1/data/info.pb
I20260812 09:45:16.650055 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 17031
I20260812 09:45:16.650082 17254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.2:35267 every 8 connection(s)
I20260812 09:45:16.650187 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/minicluster-data/ts-1/wal/instance
I20260812 09:45:16.650488 13708 external_mini_cluster.cc:1428] Setting key d57c9b2923e3a2390bc1c1bf4b6d320b
I20260812 09:45:16.651016 16743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:16.637177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.2:40705 (local address 127.13.99.62:37397)
0812 09:45:16.637351 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:16.637356 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:16.638655 (+ 1299us) server_negotiation.cc:407] Connection header received
0812 09:45:16.639695 (+ 1040us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:16.639698 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:16.639758 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:16.639863 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:16.641602 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:16.642311 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:16.643717 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:16.643961 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:16.644078 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:16.646945 (+ 2867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:16.646949 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:16.646952 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:16.646988 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:16.648634 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:16.649320 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:16.649324 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:16.649325 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:16.649421 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:16.649842 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:16.649857 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:16.649859 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:16.650042 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:16.650156 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:16.650692 (+ 536us) server_negotiation.cc:299] Negotiation successful
0812 09:45:16.650835 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260812 09:45:16.651952 17066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:16.638011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:37397 (local address 127.13.99.2:40705)
0812 09:45:16.638486 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:16.638538 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:16.639474 (+ 936us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:16.640077 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:16.640089 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:16.640637 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:45:16.641403 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:16.641438 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:16.642482 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:16.642486 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:16.643585 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:16.643596 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:16.644132 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:16.644136 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:16.644300 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:16.644931 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:45:16.644948 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:45:16.646768 (+ 1820us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:45:16.648812 (+ 2044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:16.648817 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:16.648851 (+ 34us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:16.649212 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:16.649559 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:16.649563 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:16.649565 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:16.649727 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:16.650157 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:45:16.650165 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:45:16.650446 (+ 281us) client_negotiation.cc:769] Sending connection context
0812 09:45:16.650648 (+ 202us) client_negotiation.cc:240] Negotiation successful
0812 09:45:16.650976 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":139,"threads_started":1}
I20260812 09:45:16.653193 17255 heartbeater.cc:347] Connected to a master server at 127.13.99.62:37397
I20260812 09:45:16.653335 17255 heartbeater.cc:464] Registering TS with master...
I20260812 09:45:16.653909 17255 heartbeater.cc:511] Master 127.13.99.62:37397 requested a full tablet report, sending...
I20260812 09:45:16.655035 16643 ts_manager.cc:194] Registered new tserver with Master: ca5cccf002874a15b99a3ec1fdea4c3f (127.13.99.2:35267)
I20260812 09:45:16.655756 16643 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.99.2@KRBTEST.COM'} at 127.13.99.2:40705
WARNING: no policy specified for kudu/127.13.99.3@KRBTEST.COM; defaulting to no policy
I20260812 09:45:16.665268 17269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:16.655667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:37397 (local address 127.13.99.2:57425)
0812 09:45:16.655973 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:16.655989 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:16.656347 (+ 358us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:16.657917 (+ 1570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:16.657919 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:16.658095 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:45:16.658191 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:16.658196 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:16.659277 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:16.659280 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:16.659917 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:16.659935 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:16.660600 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:16.660603 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:16.660673 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:16.661241 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:45:16.661253 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:45:16.661590 (+ 337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:45:16.663609 (+ 2019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:16.663612 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:16.663614 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:16.663817 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:16.664146 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:16.664149 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:16.664150 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:16.664230 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:16.664818 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:45:16.664821 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:45:16.664904 (+ 83us) client_negotiation.cc:769] Sending connection context
0812 09:45:16.664997 (+ 93us) client_negotiation.cc:240] Negotiation successful
0812 09:45:16.665110 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":91,"threads_started":1}
I20260812 09:45:16.665319 16743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:16.655791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.2:57425 (local address 127.13.99.62:37397)
0812 09:45:16.655908 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:16.655914 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:16.656227 (+ 313us) server_negotiation.cc:407] Connection header received
0812 09:45:16.656508 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:16.656512 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:16.656592 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:16.656671 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:16.658311 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:16.659129 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:16.660057 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:16.660444 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:16.660594 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:16.661734 (+ 1140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:16.661740 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:16.661744 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:16.661777 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:16.663487 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:16.663935 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:16.663942 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:16.663946 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:16.664047 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:16.664322 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:16.664325 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:16.664327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:16.664700 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:16.664811 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:16.665099 (+ 288us) server_negotiation.cc:299] Negotiation successful
0812 09:45:16.665205 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.99.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.99.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.3.keytab.
Entry for principal kudu/127.13.99.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.3.keytab.
I20260812 09:45:16.685316 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.99.3:0
--local_ip_for_outbound_sockets=127.13.99.3
--webserver_interface=127.13.99.3
--webserver_port=0
--tserver_master_addrs=127.13.99.62:37397
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.3.keytab
--principal=kudu/127.13.99.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:35367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:45:16.798504 17272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:45:16.798806 17272 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:45:16.798906 17272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:45:16.802956 17272 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:45:16.803041 17272 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:45:16.803112 17272 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260812 09:45:16.803156 17272 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 09:45:16.803224 17272 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.99.3
I20260812 09:45:16.808619 17272 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:35367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.3.keytab
--principal=kudu/127.13.99.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.99.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.99.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.99.62:37397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17272
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.99.3
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:45:16.809785 17272 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:45:16.810925 17272 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:45:16.820622 17285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:16.820622 17283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:16.820600 17281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:45:16.820832 17272 server_base.cc:1034] running on GCE node
I20260812 09:45:16.821277 17272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:45:16.821907 17272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:45:16.823067 17272 hybrid_clock.cc:648] HybridClock initialized: now 1786527916823026 us; error 52 us; skew 500 ppm
Aug 12 09:45:16 dist-test-slave-jq4t krb5kdc[16579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527916, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:45:16.825888 17272 init.cc:377] Logged in from keytab as kudu/127.13.99.3@KRBTEST.COM (short username kudu)
I20260812 09:45:16.826721 17272 webserver.cc:460] Webserver started at http://127.13.99.3:34371/ using document root <none> and password file <none>
I20260812 09:45:16.827270 17272 fs_manager.cc:359] Metadata directory not provided
I20260812 09:45:16.827328 17272 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:45:16.827572 17272 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:45:16.829732 17272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/minicluster-data/ts-2/data/instance:
uuid: "dfe813ac860f4be68c2f87dc4bcab956"
format_stamp: "Formatted at 2026-08-12 09:45:16 on dist-test-slave-jq4t"
server_key: "14dce63459af454ce69e97e169d24778"
server_key_iv: "c07047f9dfe186ca3b681f7ecb13f560"
server_key_version: "encryptionkey@0"
I20260812 09:45:16.830353 17272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/minicluster-data/ts-2/wal/instance:
uuid: "dfe813ac860f4be68c2f87dc4bcab956"
format_stamp: "Formatted at 2026-08-12 09:45:16 on dist-test-slave-jq4t"
server_key: "14dce63459af454ce69e97e169d24778"
server_key_iv: "c07047f9dfe186ca3b681f7ecb13f560"
server_key_version: "encryptionkey@0"
I20260812 09:45:16.834277 17272 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 09:45:16.836771 17295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:16.838049 17272 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 09:45:16.838209 17272 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/minicluster-data/ts-2/wal
uuid: "dfe813ac860f4be68c2f87dc4bcab956"
format_stamp: "Formatted at 2026-08-12 09:45:16 on dist-test-slave-jq4t"
server_key: "14dce63459af454ce69e97e169d24778"
server_key_iv: "c07047f9dfe186ca3b681f7ecb13f560"
server_key_version: "encryptionkey@0"
I20260812 09:45:16.838332 17272 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:45:16.859990 17272 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:45:16.863420 17272 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:45:16.863641 17272 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:45:16.865279 17272 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:45:16.865356 17272 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:16.865454 17272 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:45:16.865500 17272 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 09:45:16 dist-test-slave-jq4t krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527916, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.3@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
I20260812 09:45:16.876739 17272 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.3:35941
I20260812 09:45:16.876968 17493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.3:35941 every 8 connection(s)
I20260812 09:45:16.877874 17272 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/minicluster-data/ts-2/data/info.pb
I20260812 09:45:16.880865 16743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:16.865972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.3:55241 (local address 127.13.99.62:37397)
0812 09:45:16.866140 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:16.866148 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:16.867563 (+ 1415us) server_negotiation.cc:407] Connection header received
0812 09:45:16.869074 (+ 1511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:16.869077 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:16.869137 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:16.869214 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:16.871004 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:16.871655 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:16.873588 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:16.873802 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:16.873882 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:16.877007 (+ 3125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:16.877011 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:16.877014 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:16.877042 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:16.878745 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:16.879264 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:16.879268 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:16.879270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:16.879359 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:16.879732 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:16.879735 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:16.879737 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:16.879933 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:16.880045 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:16.880565 (+ 520us) server_negotiation.cc:299] Negotiation successful
0812 09:45:16.880690 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260812 09:45:16.881620 17312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:16.866861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:37397 (local address 127.13.99.3:55241)
0812 09:45:16.867379 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:16.867447 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:16.868802 (+ 1355us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:16.869473 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:16.869483 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:16.870037 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:45:16.870817 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:16.870845 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:16.871800 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:16.871807 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:16.873442 (+ 1635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:16.873454 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:16.873967 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:16.873971 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:16.874245 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:16.874901 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:45:16.874925 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:45:16.876775 (+ 1850us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:45:16.878865 (+ 2090us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:16.878872 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:16.878885 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:16.879149 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:16.879475 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:16.879478 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:16.879480 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:16.879623 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:16.880043 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:45:16.880050 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:45:16.880293 (+ 243us) client_negotiation.cc:769] Sending connection context
0812 09:45:16.880482 (+ 189us) client_negotiation.cc:240] Negotiation successful
0812 09:45:16.880822 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":117,"threads_started":1}
I20260812 09:45:16.882620 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 17272
I20260812 09:45:16.882776 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/minicluster-data/ts-2/wal/instance
I20260812 09:45:16.882999 17494 heartbeater.cc:347] Connected to a master server at 127.13.99.62:37397
I20260812 09:45:16.883090 13708 external_mini_cluster.cc:1428] Setting key 3ef6cc1e73856f66ccb4bdcb43f86d52
I20260812 09:45:16.883134 17494 heartbeater.cc:464] Registering TS with master...
I20260812 09:45:16.883672 17494 heartbeater.cc:511] Master 127.13.99.62:37397 requested a full tablet report, sending...
I20260812 09:45:16.884895 16645 ts_manager.cc:194] Registered new tserver with Master: dfe813ac860f4be68c2f87dc4bcab956 (127.13.99.3:35941)
I20260812 09:45:16.885663 16645 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.99.3@KRBTEST.COM'} at 127.13.99.3:55241
I20260812 09:45:16.892556 16743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:16.885679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.3:53989 (local address 127.13.99.62:37397)
0812 09:45:16.885806 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:16.885811 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:16.886080 (+ 269us) server_negotiation.cc:407] Connection header received
0812 09:45:16.886225 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:16.886230 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:16.886282 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:16.886357 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:16.886877 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:16.887381 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:16.888192 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:16.888424 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:16.888568 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:16.889603 (+ 1035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:16.889606 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:16.889609 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:16.889640 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:16.890883 (+ 1243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:16.891341 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:16.891346 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:16.891347 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:16.891431 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:16.891721 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:16.891725 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:16.891727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:16.891898 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:16.892013 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:16.892283 (+ 270us) server_negotiation.cc:299] Negotiation successful
0812 09:45:16.892390 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260812 09:45:16.892570 17507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:16.885556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:37397 (local address 127.13.99.3:53989)
0812 09:45:16.885991 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:16.886005 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:16.886106 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:16.886485 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:16.886487 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:16.886657 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:45:16.886759 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:16.886764 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:16.887507 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:16.887510 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:16.888071 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:16.888079 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:16.888580 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:16.888584 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:16.888686 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:16.889137 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:45:16.889149 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:45:16.889461 (+ 312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:45:16.891029 (+ 1568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:16.891035 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:16.891037 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:16.891237 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:16.891536 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:16.891539 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:16.891541 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:16.891628 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:16.892004 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:45:16.892007 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:45:16.892101 (+ 94us) client_negotiation.cc:769] Sending connection context
0812 09:45:16.892214 (+ 113us) client_negotiation.cc:240] Negotiation successful
0812 09:45:16.892356 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":112,"threads_started":1}
I20260812 09:45:16.897044 13708 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 09:45:16 dist-test-slave-jq4t krb5kdc[16579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 09:45:16 dist-test-slave-jq4t krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
I20260812 09:45:16.924556 16743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:16.915739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43112 (local address 127.13.99.62:37397)
0812 09:45:16.915885 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:16.915889 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:16.915901 (+ 12us) server_negotiation.cc:407] Connection header received
0812 09:45:16.915939 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:16.915941 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:16.915979 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:16.916042 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:16.916798 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:16.917343 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:16.918276 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:16.918517 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:16.918602 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:16.921104 (+ 2502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:16.921107 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:16.921109 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:16.921133 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:16.922484 (+ 1351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:16.923024 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:16.923027 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:16.923028 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:16.923256 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:16.923548 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:16.923550 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:16.923551 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:16.923824 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:16.923934 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:16.924178 (+ 244us) server_negotiation.cc:299] Negotiation successful
0812 09:45:16.924275 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260812 09:45:16.927480 16645 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43112:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 09:45:16.930732 16645 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 09:45:16.945794 17529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:16.939514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39042 (local address 127.13.99.3:35941)
0812 09:45:16.940729 (+ 1215us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:16.940743 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:16.940767 (+ 24us) server_negotiation.cc:407] Connection header received
0812 09:45:16.940867 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:16.940871 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:16.941067 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:16.941231 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:45:16.941588 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:16.942766 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:16.944457 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:16.945072 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:16.945221 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:16.945418 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:16.945535 (+ 117us) server_negotiation.cc:299] Negotiation successful
0812 09:45:16.945636 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1101,"thread_start_us":122,"threads_started":1}
I20260812 09:45:16.945804 17524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:16.939229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.3:35941 (local address 127.0.0.1:39042)
0812 09:45:16.939968 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:16.940023 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:16.940157 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:16.941227 (+ 1070us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:16.941231 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:16.941250 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:45:16.941371 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:16.941378 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:16.942913 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:16.942918 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:16.944321 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:16.944332 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:16.945214 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:16.945218 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:16.945346 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:16.945395 (+ 49us) client_negotiation.cc:769] Sending connection context
0812 09:45:16.945507 (+ 112us) client_negotiation.cc:240] Negotiation successful
0812 09:45:16.945631 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"mutex_wait_us":107,"thread_start_us":127,"threads_started":1}
I20260812 09:45:16.946118 17531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:16.939529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.1:42111 (local address 127.0.0.1:44768)
0812 09:45:16.940199 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:16.940213 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:16.940325 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:16.941021 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:16.941027 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:16.941051 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:45:16.941225 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:16.941233 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:16.942747 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:16.942751 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:16.944357 (+ 1606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:16.944369 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:16.945688 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:16.945691 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:16.945784 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:16.945809 (+ 25us) client_negotiation.cc:769] Sending connection context
0812 09:45:16.945896 (+ 87us) client_negotiation.cc:240] Negotiation successful
0812 09:45:16.945975 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"mutex_wait_us":22,"thread_start_us":95,"threads_started":1}
I20260812 09:45:16.946631 17533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:16.940127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44768 (local address 127.13.99.1:42111)
0812 09:45:16.940492 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:16.940500 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:16.940526 (+ 26us) server_negotiation.cc:407] Connection header received
0812 09:45:16.940613 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:16.940620 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:16.940852 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:16.941017 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:45:16.941381 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:16.942622 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:16.944530 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:16.945365 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:16.945483 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:16.945626 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:16.946392 (+ 766us) server_negotiation.cc:299] Negotiation successful
0812 09:45:16.946503 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":98,"threads_started":1}
I20260812 09:45:16.948835 17527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:16.939229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.2:35267 (local address 127.0.0.1:53428)
0812 09:45:16.940543 (+ 1314us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:16.940558 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:16.940640 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:16.941645 (+ 1005us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:16.941648 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:16.941665 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:45:16.941775 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:16.941781 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:16.943023 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:16.943026 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:16.945026 (+ 2000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:16.945037 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:16.947143 (+ 2106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:16.947147 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:16.947245 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:16.947267 (+ 22us) client_negotiation.cc:769] Sending connection context
0812 09:45:16.948599 (+ 1332us) client_negotiation.cc:240] Negotiation successful
0812 09:45:16.948687 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1202,"mutex_wait_us":757,"thread_start_us":108,"threads_started":1}
I20260812 09:45:16.948901 17525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:16.939422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53428 (local address 127.13.99.2:35267)
0812 09:45:16.940878 (+ 1456us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:16.940890 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:16.940913 (+ 23us) server_negotiation.cc:407] Connection header received
0812 09:45:16.940999 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:16.941005 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:16.941373 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:16.941632 (+ 259us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:45:16.941915 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:16.942887 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:16.945170 (+ 2283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:16.947012 (+ 1842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:16.947140 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:16.947301 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:16.948650 (+ 1349us) server_negotiation.cc:299] Negotiation successful
0812 09:45:16.948758 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1353,"thread_start_us":121,"threads_started":1}
I20260812 09:45:16.949091 17382 tablet_service.cc:1467] Processing CreateTablet for tablet b7cd7847901d4e53b87b2219819da6ba (DEFAULT_TABLE table=test-table [id=27de239c62a548a08ab67394ae56f125]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 09:45:16.949147 16887 tablet_service.cc:1467] Processing CreateTablet for tablet b7cd7847901d4e53b87b2219819da6ba (DEFAULT_TABLE table=test-table [id=27de239c62a548a08ab67394ae56f125]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 09:45:16.950280 17382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7cd7847901d4e53b87b2219819da6ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:45:16.950285 16887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7cd7847901d4e53b87b2219819da6ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:45:16.951768 17132 tablet_service.cc:1467] Processing CreateTablet for tablet b7cd7847901d4e53b87b2219819da6ba (DEFAULT_TABLE table=test-table [id=27de239c62a548a08ab67394ae56f125]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 09:45:16.952802 17132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7cd7847901d4e53b87b2219819da6ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:45:16.957429 17535 tablet_bootstrap.cc:492] T b7cd7847901d4e53b87b2219819da6ba P 9a386c33a01a49e4adf0dbcd90cf9223: Bootstrap starting.
I20260812 09:45:16.959579 17535 tablet_bootstrap.cc:654] T b7cd7847901d4e53b87b2219819da6ba P 9a386c33a01a49e4adf0dbcd90cf9223: Neither blocks nor log segments found. Creating new log.
I20260812 09:45:16.960160 17537 tablet_bootstrap.cc:492] T b7cd7847901d4e53b87b2219819da6ba P ca5cccf002874a15b99a3ec1fdea4c3f: Bootstrap starting.
I20260812 09:45:16.960801 17536 tablet_bootstrap.cc:492] T b7cd7847901d4e53b87b2219819da6ba P dfe813ac860f4be68c2f87dc4bcab956: Bootstrap starting.
I20260812 09:45:16.961073 17535 log.cc:824] T b7cd7847901d4e53b87b2219819da6ba P 9a386c33a01a49e4adf0dbcd90cf9223: Log is configured to *not* fsync() on all Append() calls
I20260812 09:45:16.962533 17537 tablet_bootstrap.cc:654] T b7cd7847901d4e53b87b2219819da6ba P ca5cccf002874a15b99a3ec1fdea4c3f: Neither blocks nor log segments found. Creating new log.
I20260812 09:45:16.962720 17536 tablet_bootstrap.cc:654] T b7cd7847901d4e53b87b2219819da6ba P dfe813ac860f4be68c2f87dc4bcab956: Neither blocks nor log segments found. Creating new log.
I20260812 09:45:16.963269 17535 tablet_bootstrap.cc:492] T b7cd7847901d4e53b87b2219819da6ba P 9a386c33a01a49e4adf0dbcd90cf9223: No bootstrap required, opened a new log
I20260812 09:45:16.963367 17537 log.cc:824] T b7cd7847901d4e53b87b2219819da6ba P ca5cccf002874a15b99a3ec1fdea4c3f: Log is configured to *not* fsync() on all Append() calls
I20260812 09:45:16.963413 17536 log.cc:824] T b7cd7847901d4e53b87b2219819da6ba P dfe813ac860f4be68c2f87dc4bcab956: Log is configured to *not* fsync() on all Append() calls
I20260812 09:45:16.963469 17535 ts_tablet_manager.cc:1397] T b7cd7847901d4e53b87b2219819da6ba P 9a386c33a01a49e4adf0dbcd90cf9223: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260812 09:45:16.965551 17537 tablet_bootstrap.cc:492] T b7cd7847901d4e53b87b2219819da6ba P ca5cccf002874a15b99a3ec1fdea4c3f: No bootstrap required, opened a new log
I20260812 09:45:16.965615 17536 tablet_bootstrap.cc:492] T b7cd7847901d4e53b87b2219819da6ba P dfe813ac860f4be68c2f87dc4bcab956: No bootstrap required, opened a new log
I20260812 09:45:16.965760 17537 ts_tablet_manager.cc:1397] T b7cd7847901d4e53b87b2219819da6ba P ca5cccf002874a15b99a3ec1fdea4c3f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260812 09:45:16.965790 17536 ts_tablet_manager.cc:1397] T b7cd7847901d4e53b87b2219819da6ba P dfe813ac860f4be68c2f87dc4bcab956: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260812 09:45:16.968245 17535 raft_consensus.cc:348] T b7cd7847901d4e53b87b2219819da6ba P 9a386c33a01a49e4adf0dbcd90cf9223 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5cccf002874a15b99a3ec1fdea4c3f" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35267 } } peers { permanent_uuid: "dfe813ac860f4be68c2f87dc4bcab956" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 35941 } } peers { permanent_uuid: "9a386c33a01a49e4adf0dbcd90cf9223" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 42111 } }
I20260812 09:45:16.968597 17535 raft_consensus.cc:374] T b7cd7847901d4e53b87b2219819da6ba P 9a386c33a01a49e4adf0dbcd90cf9223 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:45:16.968665 17535 raft_consensus.cc:729] T b7cd7847901d4e53b87b2219819da6ba P 9a386c33a01a49e4adf0dbcd90cf9223 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a386c33a01a49e4adf0dbcd90cf9223, State: Initialized, Role: FOLLOWER
I20260812 09:45:16.969050 17537 raft_consensus.cc:348] T b7cd7847901d4e53b87b2219819da6ba P ca5cccf002874a15b99a3ec1fdea4c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5cccf002874a15b99a3ec1fdea4c3f" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35267 } } peers { permanent_uuid: "dfe813ac860f4be68c2f87dc4bcab956" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 35941 } } peers { permanent_uuid: "9a386c33a01a49e4adf0dbcd90cf9223" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 42111 } }
I20260812 09:45:16.969167 17535 consensus_queue.cc:260] T b7cd7847901d4e53b87b2219819da6ba P 9a386c33a01a49e4adf0dbcd90cf9223 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5cccf002874a15b99a3ec1fdea4c3f" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35267 } } peers { permanent_uuid: "dfe813ac860f4be68c2f87dc4bcab956" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 35941 } } peers { permanent_uuid: "9a386c33a01a49e4adf0dbcd90cf9223" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 42111 } }
I20260812 09:45:16.969302 17537 raft_consensus.cc:374] T b7cd7847901d4e53b87b2219819da6ba P ca5cccf002874a15b99a3ec1fdea4c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:45:16.969375 17537 raft_consensus.cc:729] T b7cd7847901d4e53b87b2219819da6ba P ca5cccf002874a15b99a3ec1fdea4c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca5cccf002874a15b99a3ec1fdea4c3f, State: Initialized, Role: FOLLOWER
I20260812 09:45:16.969847 17537 consensus_queue.cc:260] T b7cd7847901d4e53b87b2219819da6ba P ca5cccf002874a15b99a3ec1fdea4c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5cccf002874a15b99a3ec1fdea4c3f" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35267 } } peers { permanent_uuid: "dfe813ac860f4be68c2f87dc4bcab956" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 35941 } } peers { permanent_uuid: "9a386c33a01a49e4adf0dbcd90cf9223" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 42111 } }
I20260812 09:45:16.969923 17012 heartbeater.cc:503] Master 127.13.99.62:37397 was elected leader, sending a full tablet report...
I20260812 09:45:16.970275 17535 ts_tablet_manager.cc:1428] T b7cd7847901d4e53b87b2219819da6ba P 9a386c33a01a49e4adf0dbcd90cf9223: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260812 09:45:16.970585 17255 heartbeater.cc:503] Master 127.13.99.62:37397 was elected leader, sending a full tablet report...
I20260812 09:45:16.970850 17537 ts_tablet_manager.cc:1428] T b7cd7847901d4e53b87b2219819da6ba P ca5cccf002874a15b99a3ec1fdea4c3f: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260812 09:45:16.969703 17536 raft_consensus.cc:348] T b7cd7847901d4e53b87b2219819da6ba P dfe813ac860f4be68c2f87dc4bcab956 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5cccf002874a15b99a3ec1fdea4c3f" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35267 } } peers { permanent_uuid: "dfe813ac860f4be68c2f87dc4bcab956" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 35941 } } peers { permanent_uuid: "9a386c33a01a49e4adf0dbcd90cf9223" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 42111 } }
I20260812 09:45:16.971292 17536 raft_consensus.cc:374] T b7cd7847901d4e53b87b2219819da6ba P dfe813ac860f4be68c2f87dc4bcab956 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:45:16.971382 17536 raft_consensus.cc:729] T b7cd7847901d4e53b87b2219819da6ba P dfe813ac860f4be68c2f87dc4bcab956 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfe813ac860f4be68c2f87dc4bcab956, State: Initialized, Role: FOLLOWER
I20260812 09:45:16.971773 17536 consensus_queue.cc:260] T b7cd7847901d4e53b87b2219819da6ba P dfe813ac860f4be68c2f87dc4bcab956 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5cccf002874a15b99a3ec1fdea4c3f" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35267 } } peers { permanent_uuid: "dfe813ac860f4be68c2f87dc4bcab956" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 35941 } } peers { permanent_uuid: "9a386c33a01a49e4adf0dbcd90cf9223" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 42111 } }
I20260812 09:45:16.973112 17494 heartbeater.cc:503] Master 127.13.99.62:37397 was elected leader, sending a full tablet report...
I20260812 09:45:16.973484 17536 ts_tablet_manager.cc:1428] T b7cd7847901d4e53b87b2219819da6ba P dfe813ac860f4be68c2f87dc4bcab956: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20260812 09:45:17.100069 17549 raft_consensus.cc:482] T b7cd7847901d4e53b87b2219819da6ba P dfe813ac860f4be68c2f87dc4bcab956 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 09:45:17.100289 17549 raft_consensus.cc:504] T b7cd7847901d4e53b87b2219819da6ba P dfe813ac860f4be68c2f87dc4bcab956 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5cccf002874a15b99a3ec1fdea4c3f" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35267 } } peers { permanent_uuid: "dfe813ac860f4be68c2f87dc4bcab956" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 35941 } } peers { permanent_uuid: "9a386c33a01a49e4adf0dbcd90cf9223" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 42111 } }
I20260812 09:45:17.101562 17549 leader_election.cc:288] T b7cd7847901d4e53b87b2219819da6ba P dfe813ac860f4be68c2f87dc4bcab956 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ca5cccf002874a15b99a3ec1fdea4c3f (127.13.99.2:35267), 9a386c33a01a49e4adf0dbcd90cf9223 (127.13.99.1:42111)
I20260812 09:45:17.105391 17312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:17.101830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.1:42111 (local address 127.13.99.3:56861)
0812 09:45:17.102030 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:17.102046 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:17.102145 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:17.102414 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:17.102417 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:17.102436 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:45:17.102560 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:17.102565 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:17.103320 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:17.103325 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:17.104136 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:17.104145 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:17.104932 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:17.104935 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:17.105049 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:17.105078 (+ 29us) client_negotiation.cc:769] Sending connection context
0812 09:45:17.105167 (+ 89us) client_negotiation.cc:240] Negotiation successful
0812 09:45:17.105249 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127}
I20260812 09:45:17.105384 17533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:17.101976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.3:56861 (local address 127.13.99.1:42111)
0812 09:45:17.102119 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:17.102123 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:17.102141 (+ 18us) server_negotiation.cc:407] Connection header received
0812 09:45:17.102244 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:17.102246 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:17.102291 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:17.102392 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:45:17.102676 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:17.103205 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:17.104269 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:17.104783 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:17.104924 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:17.104995 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:17.105198 (+ 203us) server_negotiation.cc:299] Negotiation successful
0812 09:45:17.105272 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260812 09:45:17.105916 17550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:17.101830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.2:35267 (local address 127.13.99.3:38477)
0812 09:45:17.102259 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:17.102270 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:17.102382 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:17.102649 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:17.102652 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:17.102667 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:45:17.103271 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:17.103279 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:17.104024 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:17.104027 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:17.104827 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:17.104836 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:17.105492 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:17.105496 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:17.105598 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:17.105622 (+ 24us) client_negotiation.cc:769] Sending connection context
0812 09:45:17.105705 (+ 83us) client_negotiation.cc:240] Negotiation successful
0812 09:45:17.105775 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"mutex_wait_us":56,"thread_start_us":139,"threads_started":1}
I20260812 09:45:17.105927 17525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:17.101892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.3:38477 (local address 127.13.99.2:35267)
0812 09:45:17.102032 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:17.102038 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:17.102353 (+ 315us) server_negotiation.cc:407] Connection header received
0812 09:45:17.102495 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:17.102500 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:17.102550 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:17.102638 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:45:17.103404 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:17.103902 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:17.104948 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:17.105347 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:17.105487 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:17.105544 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:17.105752 (+ 208us) server_negotiation.cc:299] Negotiation successful
0812 09:45:17.105823 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260812 09:45:17.106199 16935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7cd7847901d4e53b87b2219819da6ba" candidate_uuid: "dfe813ac860f4be68c2f87dc4bcab956" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a386c33a01a49e4adf0dbcd90cf9223" is_pre_election: true
I20260812 09:45:17.106536 16935 raft_consensus.cc:2454] T b7cd7847901d4e53b87b2219819da6ba P 9a386c33a01a49e4adf0dbcd90cf9223 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfe813ac860f4be68c2f87dc4bcab956 in term 0.
I20260812 09:45:17.106583 17170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7cd7847901d4e53b87b2219819da6ba" candidate_uuid: "dfe813ac860f4be68c2f87dc4bcab956" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca5cccf002874a15b99a3ec1fdea4c3f" is_pre_election: true
I20260812 09:45:17.106848 17170 raft_consensus.cc:2454] T b7cd7847901d4e53b87b2219819da6ba P ca5cccf002874a15b99a3ec1fdea4c3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfe813ac860f4be68c2f87dc4bcab956 in term 0.
I20260812 09:45:17.107095 17304 leader_election.cc:302] T b7cd7847901d4e53b87b2219819da6ba P dfe813ac860f4be68c2f87dc4bcab956 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a386c33a01a49e4adf0dbcd90cf9223, dfe813ac860f4be68c2f87dc4bcab956; no voters:
I20260812 09:45:17.107378 17549 raft_consensus.cc:2781] T b7cd7847901d4e53b87b2219819da6ba P dfe813ac860f4be68c2f87dc4bcab956 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 09:45:17.107481 17549 raft_consensus.cc:482] T b7cd7847901d4e53b87b2219819da6ba P dfe813ac860f4be68c2f87dc4bcab956 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 09:45:17.107543 17549 raft_consensus.cc:3037] T b7cd7847901d4e53b87b2219819da6ba P dfe813ac860f4be68c2f87dc4bcab956 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:45:17.108626 17549 raft_consensus.cc:504] T b7cd7847901d4e53b87b2219819da6ba P dfe813ac860f4be68c2f87dc4bcab956 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5cccf002874a15b99a3ec1fdea4c3f" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35267 } } peers { permanent_uuid: "dfe813ac860f4be68c2f87dc4bcab956" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 35941 } } peers { permanent_uuid: "9a386c33a01a49e4adf0dbcd90cf9223" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 42111 } }
I20260812 09:45:17.108994 17549 leader_election.cc:288] T b7cd7847901d4e53b87b2219819da6ba P dfe813ac860f4be68c2f87dc4bcab956 [CANDIDATE]: Term 1 election: Requested vote from peers ca5cccf002874a15b99a3ec1fdea4c3f (127.13.99.2:35267), 9a386c33a01a49e4adf0dbcd90cf9223 (127.13.99.1:42111)
I20260812 09:45:17.109457 17170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7cd7847901d4e53b87b2219819da6ba" candidate_uuid: "dfe813ac860f4be68c2f87dc4bcab956" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca5cccf002874a15b99a3ec1fdea4c3f"
I20260812 09:45:17.109596 17170 raft_consensus.cc:3037] T b7cd7847901d4e53b87b2219819da6ba P ca5cccf002874a15b99a3ec1fdea4c3f [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:45:17.109670 16935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7cd7847901d4e53b87b2219819da6ba" candidate_uuid: "dfe813ac860f4be68c2f87dc4bcab956" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a386c33a01a49e4adf0dbcd90cf9223"
I20260812 09:45:17.109833 16935 raft_consensus.cc:3037] T b7cd7847901d4e53b87b2219819da6ba P 9a386c33a01a49e4adf0dbcd90cf9223 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:45:17.110867 17170 raft_consensus.cc:2454] T b7cd7847901d4e53b87b2219819da6ba P ca5cccf002874a15b99a3ec1fdea4c3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfe813ac860f4be68c2f87dc4bcab956 in term 1.
I20260812 09:45:17.111073 16935 raft_consensus.cc:2454] T b7cd7847901d4e53b87b2219819da6ba P 9a386c33a01a49e4adf0dbcd90cf9223 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfe813ac860f4be68c2f87dc4bcab956 in term 1.
I20260812 09:45:17.111192 17307 leader_election.cc:302] T b7cd7847901d4e53b87b2219819da6ba P dfe813ac860f4be68c2f87dc4bcab956 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ca5cccf002874a15b99a3ec1fdea4c3f, dfe813ac860f4be68c2f87dc4bcab956; no voters:
I20260812 09:45:17.111402 17549 raft_consensus.cc:2781] T b7cd7847901d4e53b87b2219819da6ba P dfe813ac860f4be68c2f87dc4bcab956 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 09:45:17.111649 17549 raft_consensus.cc:686] T b7cd7847901d4e53b87b2219819da6ba P dfe813ac860f4be68c2f87dc4bcab956 [term 1 LEADER]: Becoming Leader. State: Replica: dfe813ac860f4be68c2f87dc4bcab956, State: Running, Role: LEADER
I20260812 09:45:17.111999 17549 consensus_queue.cc:237] T b7cd7847901d4e53b87b2219819da6ba P dfe813ac860f4be68c2f87dc4bcab956 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5cccf002874a15b99a3ec1fdea4c3f" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35267 } } peers { permanent_uuid: "dfe813ac860f4be68c2f87dc4bcab956" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 35941 } } peers { permanent_uuid: "9a386c33a01a49e4adf0dbcd90cf9223" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 42111 } }
I20260812 09:45:17.115478 16642 catalog_manager.cc:5495] T b7cd7847901d4e53b87b2219819da6ba P dfe813ac860f4be68c2f87dc4bcab956 reported cstate change: term changed from 0 to 1, leader changed from <none> to dfe813ac860f4be68c2f87dc4bcab956 (127.13.99.3). New cstate: current_term: 1 leader_uuid: "dfe813ac860f4be68c2f87dc4bcab956" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5cccf002874a15b99a3ec1fdea4c3f" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfe813ac860f4be68c2f87dc4bcab956" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 35941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a386c33a01a49e4adf0dbcd90cf9223" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 42111 } health_report { overall_health: UNKNOWN } } }
W20260812 09:45:17.128921 17497 tablet.cc:2369] T b7cd7847901d4e53b87b2219819da6ba P dfe813ac860f4be68c2f87dc4bcab956: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 09:45:17.150245 17256 tablet.cc:2369] T b7cd7847901d4e53b87b2219819da6ba P ca5cccf002874a15b99a3ec1fdea4c3f: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 12 09:45:17 dist-test-slave-jq4t krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527916, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.62@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
I20260812 09:45:17.158869 17566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:17.150151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:37397 (local address 127.0.0.1:43118)
0812 09:45:17.150574 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:17.150590 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:17.150719 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:17.151003 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:17.151006 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:17.151386 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:45:17.151495 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:17.151501 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:17.152228 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:17.152231 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:17.152857 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:17.152866 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:17.153342 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:17.153345 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:17.153417 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:17.154018 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:45:17.154030 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:45:17.155487 (+ 1457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:45:17.157392 (+ 1905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:17.157400 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:17.157405 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:17.157637 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:17.157962 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:17.157964 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:17.157966 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:17.158044 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:17.158413 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:45:17.158419 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:45:17.158525 (+ 106us) client_negotiation.cc:769] Sending connection context
0812 09:45:17.158617 (+ 92us) client_negotiation.cc:240] Negotiation successful
0812 09:45:17.158724 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":124,"threads_started":1}
I20260812 09:45:17.159092 16743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:17.150249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43118 (local address 127.13.99.62:37397)
0812 09:45:17.150401 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:17.150406 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:17.150680 (+ 274us) server_negotiation.cc:407] Connection header received
0812 09:45:17.150839 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:17.150842 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:17.150886 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:17.150971 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:17.151603 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:17.152131 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:17.152981 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:17.153192 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:17.153275 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:17.155621 (+ 2346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:17.155624 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:17.155626 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:17.155648 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:17.157273 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:17.157765 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:17.157768 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:17.157769 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:17.157855 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:17.158129 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:17.158131 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:17.158132 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:17.158319 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:17.158397 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:17.158690 (+ 293us) server_negotiation.cc:299] Negotiation successful
0812 09:45:17.158794 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260812 09:45:17.162948 16642 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.13.99.62@KRBTEST.COM'} at 127.0.0.1:43118:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260812 09:45:17.163622 16642 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 09:45:17.169349 16887 tablet_service.cc:1467] Processing CreateTablet for tablet eb70bad139084dd5945072974aedae81 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=dd35d6d395474fccb0ef4cdcea0de700]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 09:45:17.169366 17132 tablet_service.cc:1467] Processing CreateTablet for tablet eb70bad139084dd5945072974aedae81 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=dd35d6d395474fccb0ef4cdcea0de700]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 09:45:17.169466 17382 tablet_service.cc:1467] Processing CreateTablet for tablet eb70bad139084dd5945072974aedae81 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=dd35d6d395474fccb0ef4cdcea0de700]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 09:45:17.170161 16887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb70bad139084dd5945072974aedae81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:45:17.170161 17132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb70bad139084dd5945072974aedae81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:45:17.170198 17382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb70bad139084dd5945072974aedae81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:45:17.173292 17535 tablet_bootstrap.cc:492] T eb70bad139084dd5945072974aedae81 P 9a386c33a01a49e4adf0dbcd90cf9223: Bootstrap starting.
I20260812 09:45:17.173302 17537 tablet_bootstrap.cc:492] T eb70bad139084dd5945072974aedae81 P ca5cccf002874a15b99a3ec1fdea4c3f: Bootstrap starting.
I20260812 09:45:17.173398 17536 tablet_bootstrap.cc:492] T eb70bad139084dd5945072974aedae81 P dfe813ac860f4be68c2f87dc4bcab956: Bootstrap starting.
I20260812 09:45:17.174332 17536 tablet_bootstrap.cc:654] T eb70bad139084dd5945072974aedae81 P dfe813ac860f4be68c2f87dc4bcab956: Neither blocks nor log segments found. Creating new log.
I20260812 09:45:17.174333 17535 tablet_bootstrap.cc:654] T eb70bad139084dd5945072974aedae81 P 9a386c33a01a49e4adf0dbcd90cf9223: Neither blocks nor log segments found. Creating new log.
I20260812 09:45:17.174333 17537 tablet_bootstrap.cc:654] T eb70bad139084dd5945072974aedae81 P ca5cccf002874a15b99a3ec1fdea4c3f: Neither blocks nor log segments found. Creating new log.
I20260812 09:45:17.175702 17537 tablet_bootstrap.cc:492] T eb70bad139084dd5945072974aedae81 P ca5cccf002874a15b99a3ec1fdea4c3f: No bootstrap required, opened a new log
I20260812 09:45:17.175797 17537 ts_tablet_manager.cc:1397] T eb70bad139084dd5945072974aedae81 P ca5cccf002874a15b99a3ec1fdea4c3f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 09:45:17.175895 17536 tablet_bootstrap.cc:492] T eb70bad139084dd5945072974aedae81 P dfe813ac860f4be68c2f87dc4bcab956: No bootstrap required, opened a new log
I20260812 09:45:17.176010 17536 ts_tablet_manager.cc:1397] T eb70bad139084dd5945072974aedae81 P dfe813ac860f4be68c2f87dc4bcab956: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
W20260812 09:45:17.176573 17015 tablet.cc:2369] T b7cd7847901d4e53b87b2219819da6ba P 9a386c33a01a49e4adf0dbcd90cf9223: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 09:45:17.176517 17536 raft_consensus.cc:348] T eb70bad139084dd5945072974aedae81 P dfe813ac860f4be68c2f87dc4bcab956 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a386c33a01a49e4adf0dbcd90cf9223" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 42111 } } peers { permanent_uuid: "ca5cccf002874a15b99a3ec1fdea4c3f" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35267 } } peers { permanent_uuid: "dfe813ac860f4be68c2f87dc4bcab956" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 35941 } }
I20260812 09:45:17.176581 17537 raft_consensus.cc:348] T eb70bad139084dd5945072974aedae81 P ca5cccf002874a15b99a3ec1fdea4c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a386c33a01a49e4adf0dbcd90cf9223" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 42111 } } peers { permanent_uuid: "ca5cccf002874a15b99a3ec1fdea4c3f" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35267 } } peers { permanent_uuid: "dfe813ac860f4be68c2f87dc4bcab956" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 35941 } }
I20260812 09:45:17.176666 17536 raft_consensus.cc:374] T eb70bad139084dd5945072974aedae81 P dfe813ac860f4be68c2f87dc4bcab956 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:45:17.176687 17537 raft_consensus.cc:374] T eb70bad139084dd5945072974aedae81 P ca5cccf002874a15b99a3ec1fdea4c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:45:17.176712 17536 raft_consensus.cc:729] T eb70bad139084dd5945072974aedae81 P dfe813ac860f4be68c2f87dc4bcab956 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfe813ac860f4be68c2f87dc4bcab956, State: Initialized, Role: FOLLOWER
I20260812 09:45:17.176733 17537 raft_consensus.cc:729] T eb70bad139084dd5945072974aedae81 P ca5cccf002874a15b99a3ec1fdea4c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca5cccf002874a15b99a3ec1fdea4c3f, State: Initialized, Role: FOLLOWER
I20260812 09:45:17.176867 17536 consensus_queue.cc:260] T eb70bad139084dd5945072974aedae81 P dfe813ac860f4be68c2f87dc4bcab956 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a386c33a01a49e4adf0dbcd90cf9223" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 42111 } } peers { permanent_uuid: "ca5cccf002874a15b99a3ec1fdea4c3f" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35267 } } peers { permanent_uuid: "dfe813ac860f4be68c2f87dc4bcab956" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 35941 } }
I20260812 09:45:17.176887 17537 consensus_queue.cc:260] T eb70bad139084dd5945072974aedae81 P ca5cccf002874a15b99a3ec1fdea4c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a386c33a01a49e4adf0dbcd90cf9223" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 42111 } } peers { permanent_uuid: "ca5cccf002874a15b99a3ec1fdea4c3f" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35267 } } peers { permanent_uuid: "dfe813ac860f4be68c2f87dc4bcab956" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 35941 } }
I20260812 09:45:17.177142 17537 ts_tablet_manager.cc:1428] T eb70bad139084dd5945072974aedae81 P ca5cccf002874a15b99a3ec1fdea4c3f: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260812 09:45:17.177155 17536 ts_tablet_manager.cc:1428] T eb70bad139084dd5945072974aedae81 P dfe813ac860f4be68c2f87dc4bcab956: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 09:45:17.177121 17549 tablet_replica.cc:439] T eb70bad139084dd5945072974aedae81 P dfe813ac860f4be68c2f87dc4bcab956: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a386c33a01a49e4adf0dbcd90cf9223" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 42111 } } peers { permanent_uuid: "ca5cccf002874a15b99a3ec1fdea4c3f" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35267 } } peers { permanent_uuid: "dfe813ac860f4be68c2f87dc4bcab956" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 35941 } } }
I20260812 09:45:17.177249 17549 tablet_replica.cc:442] T eb70bad139084dd5945072974aedae81 P dfe813ac860f4be68c2f87dc4bcab956: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 09:45:17.177466 17548 tablet_replica.cc:439] T eb70bad139084dd5945072974aedae81 P ca5cccf002874a15b99a3ec1fdea4c3f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a386c33a01a49e4adf0dbcd90cf9223" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 42111 } } peers { permanent_uuid: "ca5cccf002874a15b99a3ec1fdea4c3f" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35267 } } peers { permanent_uuid: "dfe813ac860f4be68c2f87dc4bcab956" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 35941 } } }
I20260812 09:45:17.177614 17548 tablet_replica.cc:442] T eb70bad139084dd5945072974aedae81 P ca5cccf002874a15b99a3ec1fdea4c3f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 09:45:17.178166 17535 tablet_bootstrap.cc:492] T eb70bad139084dd5945072974aedae81 P 9a386c33a01a49e4adf0dbcd90cf9223: No bootstrap required, opened a new log
I20260812 09:45:17.178258 17535 ts_tablet_manager.cc:1397] T eb70bad139084dd5945072974aedae81 P 9a386c33a01a49e4adf0dbcd90cf9223: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260812 09:45:17.178772 17535 raft_consensus.cc:348] T eb70bad139084dd5945072974aedae81 P 9a386c33a01a49e4adf0dbcd90cf9223 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a386c33a01a49e4adf0dbcd90cf9223" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 42111 } } peers { permanent_uuid: "ca5cccf002874a15b99a3ec1fdea4c3f" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35267 } } peers { permanent_uuid: "dfe813ac860f4be68c2f87dc4bcab956" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 35941 } }
I20260812 09:45:17.178925 17535 raft_consensus.cc:374] T eb70bad139084dd5945072974aedae81 P 9a386c33a01a49e4adf0dbcd90cf9223 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:45:17.178958 17535 raft_consensus.cc:729] T eb70bad139084dd5945072974aedae81 P 9a386c33a01a49e4adf0dbcd90cf9223 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a386c33a01a49e4adf0dbcd90cf9223, State: Initialized, Role: FOLLOWER
I20260812 09:45:17.179092 17535 consensus_queue.cc:260] T eb70bad139084dd5945072974aedae81 P 9a386c33a01a49e4adf0dbcd90cf9223 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a386c33a01a49e4adf0dbcd90cf9223" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 42111 } } peers { permanent_uuid: "ca5cccf002874a15b99a3ec1fdea4c3f" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35267 } } peers { permanent_uuid: "dfe813ac860f4be68c2f87dc4bcab956" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 35941 } }
I20260812 09:45:17.179382 17545 tablet_replica.cc:439] T eb70bad139084dd5945072974aedae81 P 9a386c33a01a49e4adf0dbcd90cf9223: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a386c33a01a49e4adf0dbcd90cf9223" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 42111 } } peers { permanent_uuid: "ca5cccf002874a15b99a3ec1fdea4c3f" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35267 } } peers { permanent_uuid: "dfe813ac860f4be68c2f87dc4bcab956" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 35941 } } }
I20260812 09:45:17.179383 17535 ts_tablet_manager.cc:1428] T eb70bad139084dd5945072974aedae81 P 9a386c33a01a49e4adf0dbcd90cf9223: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260812 09:45:17.179538 17545 tablet_replica.cc:442] T eb70bad139084dd5945072974aedae81 P 9a386c33a01a49e4adf0dbcd90cf9223: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 09:45:17.287034 17549 raft_consensus.cc:482] T eb70bad139084dd5945072974aedae81 P dfe813ac860f4be68c2f87dc4bcab956 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 09:45:17.287171 17549 raft_consensus.cc:504] T eb70bad139084dd5945072974aedae81 P dfe813ac860f4be68c2f87dc4bcab956 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a386c33a01a49e4adf0dbcd90cf9223" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 42111 } } peers { permanent_uuid: "ca5cccf002874a15b99a3ec1fdea4c3f" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35267 } } peers { permanent_uuid: "dfe813ac860f4be68c2f87dc4bcab956" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 35941 } }
I20260812 09:45:17.287676 17549 leader_election.cc:288] T eb70bad139084dd5945072974aedae81 P dfe813ac860f4be68c2f87dc4bcab956 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a386c33a01a49e4adf0dbcd90cf9223 (127.13.99.1:42111), ca5cccf002874a15b99a3ec1fdea4c3f (127.13.99.2:35267)
I20260812 09:45:17.288214 17170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb70bad139084dd5945072974aedae81" candidate_uuid: "dfe813ac860f4be68c2f87dc4bcab956" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca5cccf002874a15b99a3ec1fdea4c3f" is_pre_election: true
I20260812 09:45:17.288244 16935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb70bad139084dd5945072974aedae81" candidate_uuid: "dfe813ac860f4be68c2f87dc4bcab956" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a386c33a01a49e4adf0dbcd90cf9223" is_pre_election: true
I20260812 09:45:17.288390 17170 raft_consensus.cc:2454] T eb70bad139084dd5945072974aedae81 P ca5cccf002874a15b99a3ec1fdea4c3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfe813ac860f4be68c2f87dc4bcab956 in term 0.
I20260812 09:45:17.288416 16935 raft_consensus.cc:2454] T eb70bad139084dd5945072974aedae81 P 9a386c33a01a49e4adf0dbcd90cf9223 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfe813ac860f4be68c2f87dc4bcab956 in term 0.
I20260812 09:45:17.288741 17307 leader_election.cc:302] T eb70bad139084dd5945072974aedae81 P dfe813ac860f4be68c2f87dc4bcab956 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ca5cccf002874a15b99a3ec1fdea4c3f, dfe813ac860f4be68c2f87dc4bcab956; no voters:
I20260812 09:45:17.288938 17549 raft_consensus.cc:2781] T eb70bad139084dd5945072974aedae81 P dfe813ac860f4be68c2f87dc4bcab956 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 09:45:17.289000 17549 raft_consensus.cc:482] T eb70bad139084dd5945072974aedae81 P dfe813ac860f4be68c2f87dc4bcab956 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 09:45:17.289050 17549 raft_consensus.cc:3037] T eb70bad139084dd5945072974aedae81 P dfe813ac860f4be68c2f87dc4bcab956 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:45:17.290058 17549 raft_consensus.cc:504] T eb70bad139084dd5945072974aedae81 P dfe813ac860f4be68c2f87dc4bcab956 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a386c33a01a49e4adf0dbcd90cf9223" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 42111 } } peers { permanent_uuid: "ca5cccf002874a15b99a3ec1fdea4c3f" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35267 } } peers { permanent_uuid: "dfe813ac860f4be68c2f87dc4bcab956" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 35941 } }
I20260812 09:45:17.290450 17549 leader_election.cc:288] T eb70bad139084dd5945072974aedae81 P dfe813ac860f4be68c2f87dc4bcab956 [CANDIDATE]: Term 1 election: Requested vote from peers 9a386c33a01a49e4adf0dbcd90cf9223 (127.13.99.1:42111), ca5cccf002874a15b99a3ec1fdea4c3f (127.13.99.2:35267)
I20260812 09:45:17.290824 17170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb70bad139084dd5945072974aedae81" candidate_uuid: "dfe813ac860f4be68c2f87dc4bcab956" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca5cccf002874a15b99a3ec1fdea4c3f"
I20260812 09:45:17.291004 17170 raft_consensus.cc:3037] T eb70bad139084dd5945072974aedae81 P ca5cccf002874a15b99a3ec1fdea4c3f [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:45:17.292174 17170 raft_consensus.cc:2454] T eb70bad139084dd5945072974aedae81 P ca5cccf002874a15b99a3ec1fdea4c3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfe813ac860f4be68c2f87dc4bcab956 in term 1.
I20260812 09:45:17.292470 17307 leader_election.cc:302] T eb70bad139084dd5945072974aedae81 P dfe813ac860f4be68c2f87dc4bcab956 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ca5cccf002874a15b99a3ec1fdea4c3f, dfe813ac860f4be68c2f87dc4bcab956; no voters:
I20260812 09:45:17.292452 16935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb70bad139084dd5945072974aedae81" candidate_uuid: "dfe813ac860f4be68c2f87dc4bcab956" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a386c33a01a49e4adf0dbcd90cf9223"
I20260812 09:45:17.292656 16935 raft_consensus.cc:3037] T eb70bad139084dd5945072974aedae81 P 9a386c33a01a49e4adf0dbcd90cf9223 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:45:17.292675 17549 raft_consensus.cc:2781] T eb70bad139084dd5945072974aedae81 P dfe813ac860f4be68c2f87dc4bcab956 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 09:45:17.292774 17549 raft_consensus.cc:686] T eb70bad139084dd5945072974aedae81 P dfe813ac860f4be68c2f87dc4bcab956 [term 1 LEADER]: Becoming Leader. State: Replica: dfe813ac860f4be68c2f87dc4bcab956, State: Running, Role: LEADER
I20260812 09:45:17.292917 17549 consensus_queue.cc:237] T eb70bad139084dd5945072974aedae81 P dfe813ac860f4be68c2f87dc4bcab956 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a386c33a01a49e4adf0dbcd90cf9223" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 42111 } } peers { permanent_uuid: "ca5cccf002874a15b99a3ec1fdea4c3f" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35267 } } peers { permanent_uuid: "dfe813ac860f4be68c2f87dc4bcab956" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 35941 } }
I20260812 09:45:17.293664 17553 tablet_replica.cc:439] T eb70bad139084dd5945072974aedae81 P dfe813ac860f4be68c2f87dc4bcab956: TxnStatusTablet state changed. Reason: New leader dfe813ac860f4be68c2f87dc4bcab956. Latest consensus state: current_term: 1 leader_uuid: "dfe813ac860f4be68c2f87dc4bcab956" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a386c33a01a49e4adf0dbcd90cf9223" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 42111 } } peers { permanent_uuid: "ca5cccf002874a15b99a3ec1fdea4c3f" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35267 } } peers { permanent_uuid: "dfe813ac860f4be68c2f87dc4bcab956" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 35941 } } }
I20260812 09:45:17.293785 17553 tablet_replica.cc:442] T eb70bad139084dd5945072974aedae81 P dfe813ac860f4be68c2f87dc4bcab956: This TxnStatusTablet replica's current role is: LEADER
I20260812 09:45:17.293789 16935 raft_consensus.cc:2454] T eb70bad139084dd5945072974aedae81 P 9a386c33a01a49e4adf0dbcd90cf9223 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfe813ac860f4be68c2f87dc4bcab956 in term 1.
I20260812 09:45:17.294147 17573 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260812 09:45:17.294428 17573 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260812 09:45:17.295094 16643 catalog_manager.cc:5495] T eb70bad139084dd5945072974aedae81 P dfe813ac860f4be68c2f87dc4bcab956 reported cstate change: term changed from 0 to 1, leader changed from <none> to dfe813ac860f4be68c2f87dc4bcab956 (127.13.99.3). New cstate: current_term: 1 leader_uuid: "dfe813ac860f4be68c2f87dc4bcab956" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a386c33a01a49e4adf0dbcd90cf9223" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 42111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca5cccf002874a15b99a3ec1fdea4c3f" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfe813ac860f4be68c2f87dc4bcab956" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 35941 } health_report { overall_health: HEALTHY } } }
I20260812 09:45:17.331764 17529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:17.327919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39058 (local address 127.13.99.3:35941)
0812 09:45:17.328051 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:17.328057 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:17.328071 (+ 14us) server_negotiation.cc:407] Connection header received
0812 09:45:17.328296 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:17.328300 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:17.328371 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:17.328481 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 09:45:17.328763 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:17.329363 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:17.330263 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:17.330490 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:17.330606 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:17.330937 (+ 331us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 09:45:17.331351 (+ 414us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 09:45:17.331484 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:17.331560 (+ 76us) server_negotiation.cc:299] Negotiation successful
0812 09:45:17.331620 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260812 09:45:17.331763 17566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:17.327787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.3:35941 (local address 127.0.0.1:39058)
0812 09:45:17.327986 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:17.328004 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:17.328128 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:17.328525 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:17.328529 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:17.328537 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 09:45:17.328647 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:17.328653 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:17.329479 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:17.329482 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:17.330116 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:17.330144 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:17.330609 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:17.330613 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:17.330708 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:17.330754 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 09:45:17.331462 (+ 708us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 09:45:17.331479 (+ 17us) client_negotiation.cc:769] Sending connection context
0812 09:45:17.331551 (+ 72us) client_negotiation.cc:240] Negotiation successful
0812 09:45:17.331639 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260812 09:45:17.337785 17170 raft_consensus.cc:1261] T eb70bad139084dd5945072974aedae81 P ca5cccf002874a15b99a3ec1fdea4c3f [term 1 FOLLOWER]: Refusing update from remote peer dfe813ac860f4be68c2f87dc4bcab956: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 09:45:17.337805 16935 raft_consensus.cc:1261] T eb70bad139084dd5945072974aedae81 P 9a386c33a01a49e4adf0dbcd90cf9223 [term 1 FOLLOWER]: Refusing update from remote peer dfe813ac860f4be68c2f87dc4bcab956: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 09:45:17.338551 17549 consensus_queue.cc:1035] T eb70bad139084dd5945072974aedae81 P dfe813ac860f4be68c2f87dc4bcab956 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5cccf002874a15b99a3ec1fdea4c3f" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 09:45:17.338816 17553 consensus_queue.cc:1035] T eb70bad139084dd5945072974aedae81 P dfe813ac860f4be68c2f87dc4bcab956 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a386c33a01a49e4adf0dbcd90cf9223" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 42111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 09:45:17.343093 17545 tablet_replica.cc:439] T eb70bad139084dd5945072974aedae81 P 9a386c33a01a49e4adf0dbcd90cf9223: TxnStatusTablet state changed. Reason: New leader dfe813ac860f4be68c2f87dc4bcab956. Latest consensus state: current_term: 1 leader_uuid: "dfe813ac860f4be68c2f87dc4bcab956" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a386c33a01a49e4adf0dbcd90cf9223" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 42111 } } peers { permanent_uuid: "ca5cccf002874a15b99a3ec1fdea4c3f" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35267 } } peers { permanent_uuid: "dfe813ac860f4be68c2f87dc4bcab956" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 35941 } } }
I20260812 09:45:17.343236 17545 tablet_replica.cc:442] T eb70bad139084dd5945072974aedae81 P 9a386c33a01a49e4adf0dbcd90cf9223: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 09:45:17.343585 17548 tablet_replica.cc:439] T eb70bad139084dd5945072974aedae81 P ca5cccf002874a15b99a3ec1fdea4c3f: TxnStatusTablet state changed. Reason: New leader dfe813ac860f4be68c2f87dc4bcab956. Latest consensus state: current_term: 1 leader_uuid: "dfe813ac860f4be68c2f87dc4bcab956" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a386c33a01a49e4adf0dbcd90cf9223" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 42111 } } peers { permanent_uuid: "ca5cccf002874a15b99a3ec1fdea4c3f" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35267 } } peers { permanent_uuid: "dfe813ac860f4be68c2f87dc4bcab956" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 35941 } } }
I20260812 09:45:17.343725 17548 tablet_replica.cc:442] T eb70bad139084dd5945072974aedae81 P ca5cccf002874a15b99a3ec1fdea4c3f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 09:45:17.346485 17549 tablet_replica.cc:439] T eb70bad139084dd5945072974aedae81 P dfe813ac860f4be68c2f87dc4bcab956: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dfe813ac860f4be68c2f87dc4bcab956" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a386c33a01a49e4adf0dbcd90cf9223" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 42111 } } peers { permanent_uuid: "ca5cccf002874a15b99a3ec1fdea4c3f" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35267 } } peers { permanent_uuid: "dfe813ac860f4be68c2f87dc4bcab956" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 35941 } } }
I20260812 09:45:17.346621 17549 tablet_replica.cc:442] T eb70bad139084dd5945072974aedae81 P dfe813ac860f4be68c2f87dc4bcab956: This TxnStatusTablet replica's current role is: LEADER
I20260812 09:45:17.346877 17578 mvcc.cc:205] Tried to move back new op lower bound from 7317618349407608832 to 7317618349233025024. Current Snapshot: MvccSnapshot[applied={T|T < 7317618349407608832}]
I20260812 09:45:17.348057 17545 tablet_replica.cc:439] T eb70bad139084dd5945072974aedae81 P 9a386c33a01a49e4adf0dbcd90cf9223: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dfe813ac860f4be68c2f87dc4bcab956" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a386c33a01a49e4adf0dbcd90cf9223" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 42111 } } peers { permanent_uuid: "ca5cccf002874a15b99a3ec1fdea4c3f" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35267 } } peers { permanent_uuid: "dfe813ac860f4be68c2f87dc4bcab956" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 35941 } } }
I20260812 09:45:17.348177 17545 tablet_replica.cc:442] T eb70bad139084dd5945072974aedae81 P 9a386c33a01a49e4adf0dbcd90cf9223: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 09:45:17.348730 17580 mvcc.cc:205] Tried to move back new op lower bound from 7317618349407608832 to 7317618349233025024. Current Snapshot: MvccSnapshot[applied={T|T < 7317618349407608832}]
I20260812 09:45:17.353470 17553 tablet_replica.cc:439] T eb70bad139084dd5945072974aedae81 P dfe813ac860f4be68c2f87dc4bcab956: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dfe813ac860f4be68c2f87dc4bcab956" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a386c33a01a49e4adf0dbcd90cf9223" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 42111 } } peers { permanent_uuid: "ca5cccf002874a15b99a3ec1fdea4c3f" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35267 } } peers { permanent_uuid: "dfe813ac860f4be68c2f87dc4bcab956" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 35941 } } }
I20260812 09:45:17.353597 17553 tablet_replica.cc:442] T eb70bad139084dd5945072974aedae81 P dfe813ac860f4be68c2f87dc4bcab956: This TxnStatusTablet replica's current role is: LEADER
I20260812 09:45:17.355684 17548 tablet_replica.cc:439] T eb70bad139084dd5945072974aedae81 P ca5cccf002874a15b99a3ec1fdea4c3f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dfe813ac860f4be68c2f87dc4bcab956" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a386c33a01a49e4adf0dbcd90cf9223" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 42111 } } peers { permanent_uuid: "ca5cccf002874a15b99a3ec1fdea4c3f" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35267 } } peers { permanent_uuid: "dfe813ac860f4be68c2f87dc4bcab956" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 35941 } } }
I20260812 09:45:17.355801 17548 tablet_replica.cc:442] T eb70bad139084dd5945072974aedae81 P ca5cccf002874a15b99a3ec1fdea4c3f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 09:45:17.358636 17529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:17.354194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39068 (local address 127.13.99.3:35941)
0812 09:45:17.354312 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:17.354317 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:17.354405 (+ 88us) server_negotiation.cc:407] Connection header received
0812 09:45:17.354585 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:17.354589 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:17.354642 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:17.354726 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 09:45:17.355009 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:17.355725 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:17.357121 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:17.357443 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:17.357548 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:17.357965 (+ 417us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 09:45:17.358051 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 09:45:17.358117 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:17.358474 (+ 357us) server_negotiation.cc:299] Negotiation successful
0812 09:45:17.358529 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260812 09:45:17.369593 17529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:17.366356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.3:53353 (local address 127.13.99.3:35941)
0812 09:45:17.366602 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:17.366605 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:17.366612 (+ 7us) server_negotiation.cc:407] Connection header received
0812 09:45:17.366636 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:17.366638 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:17.366666 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:17.366745 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 09:45:17.367003 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:17.367472 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:17.368421 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:17.368697 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:17.368804 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:17.369091 (+ 287us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 09:45:17.369193 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 09:45:17.369275 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:17.369401 (+ 126us) server_negotiation.cc:299] Negotiation successful
0812 09:45:17.369438 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201}
I20260812 09:45:17.369593 17507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:17.366267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.3:35941 (local address 127.13.99.3:53353)
0812 09:45:17.366425 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:17.366437 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:17.366505 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:17.366753 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:17.366756 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:17.366762 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 09:45:17.366891 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:17.366897 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:17.367666 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:17.367671 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:17.368281 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:17.368290 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:17.368804 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:17.368808 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:17.368900 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:17.368939 (+ 39us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 09:45:17.369290 (+ 351us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 09:45:17.369296 (+ 6us) client_negotiation.cc:769] Sending connection context
0812 09:45:17.369370 (+ 74us) client_negotiation.cc:240] Negotiation successful
0812 09:45:17.369444 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":1}
I20260812 09:45:17.376339 16935 raft_consensus.cc:1261] T b7cd7847901d4e53b87b2219819da6ba P 9a386c33a01a49e4adf0dbcd90cf9223 [term 1 FOLLOWER]: Refusing update from remote peer dfe813ac860f4be68c2f87dc4bcab956: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 09:45:17.376451 17170 raft_consensus.cc:1261] T b7cd7847901d4e53b87b2219819da6ba P ca5cccf002874a15b99a3ec1fdea4c3f [term 1 FOLLOWER]: Refusing update from remote peer dfe813ac860f4be68c2f87dc4bcab956: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 09:45:17.376751 17549 consensus_queue.cc:1035] T b7cd7847901d4e53b87b2219819da6ba P dfe813ac860f4be68c2f87dc4bcab956 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a386c33a01a49e4adf0dbcd90cf9223" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 42111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 09:45:17.377179 17549 consensus_queue.cc:1035] T b7cd7847901d4e53b87b2219819da6ba P dfe813ac860f4be68c2f87dc4bcab956 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5cccf002874a15b99a3ec1fdea4c3f" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 09:45:17.392784 17579 mvcc.cc:205] Tried to move back new op lower bound from 7317618349570146304 to 7317618348493721600. Current Snapshot: MvccSnapshot[applied={T|T < 7317618349570146304}]
I20260812 09:45:17.421500 16642 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43112:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 09:45:17.421711 16642 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43112:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 09:45:17.424319 16642 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 77dec44a5cf34f3a9c0e1dfc5ff57cd4: Sending DeleteTablet for 3 replicas of tablet b7cd7847901d4e53b87b2219819da6ba
I20260812 09:45:17.425155 17132 tablet_service.cc:1514] Processing DeleteTablet for tablet b7cd7847901d4e53b87b2219819da6ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 09:45:17 UTC) from {username='kudu', principal='kudu/127.13.99.62@KRBTEST.COM'} at 127.0.0.1:53428
I20260812 09:45:17.425714 17599 tablet_replica.cc:330] T b7cd7847901d4e53b87b2219819da6ba P ca5cccf002874a15b99a3ec1fdea4c3f: stopping tablet replica
I20260812 09:45:17.425995 16887 tablet_service.cc:1514] Processing DeleteTablet for tablet b7cd7847901d4e53b87b2219819da6ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 09:45:17 UTC) from {username='kudu', principal='kudu/127.13.99.62@KRBTEST.COM'} at 127.0.0.1:44768
I20260812 09:45:17.426200 17599 raft_consensus.cc:2229] T b7cd7847901d4e53b87b2219819da6ba P ca5cccf002874a15b99a3ec1fdea4c3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 09:45:17.426473 17599 raft_consensus.cc:2258] T b7cd7847901d4e53b87b2219819da6ba P ca5cccf002874a15b99a3ec1fdea4c3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 09:45:17.426721 17601 tablet_replica.cc:330] T b7cd7847901d4e53b87b2219819da6ba P 9a386c33a01a49e4adf0dbcd90cf9223: stopping tablet replica
I20260812 09:45:17.426980 17601 raft_consensus.cc:2229] T b7cd7847901d4e53b87b2219819da6ba P 9a386c33a01a49e4adf0dbcd90cf9223 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 09:45:17.427187 17601 raft_consensus.cc:2258] T b7cd7847901d4e53b87b2219819da6ba P 9a386c33a01a49e4adf0dbcd90cf9223 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 09:45:17.429032 17382 tablet_service.cc:1514] Processing DeleteTablet for tablet b7cd7847901d4e53b87b2219819da6ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 09:45:17 UTC) from {username='kudu', principal='kudu/127.13.99.62@KRBTEST.COM'} at 127.0.0.1:39042
I20260812 09:45:17.429327 16642 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43112:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 09:45:17.429752 16642 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 09:45:17.429870 17601 ts_tablet_manager.cc:1905] T b7cd7847901d4e53b87b2219819da6ba P 9a386c33a01a49e4adf0dbcd90cf9223: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 09:45:17.434274 17601 ts_tablet_manager.cc:1918] T b7cd7847901d4e53b87b2219819da6ba P 9a386c33a01a49e4adf0dbcd90cf9223: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 09:45:17.434382 17601 log.cc:1196] T b7cd7847901d4e53b87b2219819da6ba P 9a386c33a01a49e4adf0dbcd90cf9223: Deleting WAL directory at /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/minicluster-data/ts-0/wal/wals/b7cd7847901d4e53b87b2219819da6ba
I20260812 09:45:17.434726 17601 ts_tablet_manager.cc:1939] T b7cd7847901d4e53b87b2219819da6ba P 9a386c33a01a49e4adf0dbcd90cf9223: Deleting consensus metadata
I20260812 09:45:17.435535 16887 tablet_service.cc:1467] Processing CreateTablet for tablet 3d64a70d4516472dacd20193ef9f01c0 (DEFAULT_TABLE table=test-table [id=f08f0287acab4770a6a7f00c8f9e6f79]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 09:45:17.435762 16887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d64a70d4516472dacd20193ef9f01c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:45:17.436952 17599 ts_tablet_manager.cc:1905] T b7cd7847901d4e53b87b2219819da6ba P ca5cccf002874a15b99a3ec1fdea4c3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 09:45:17.438086 17132 tablet_service.cc:1467] Processing CreateTablet for tablet 3d64a70d4516472dacd20193ef9f01c0 (DEFAULT_TABLE table=test-table [id=f08f0287acab4770a6a7f00c8f9e6f79]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 09:45:17.438323 17132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d64a70d4516472dacd20193ef9f01c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:45:17.439776 17599 ts_tablet_manager.cc:1918] T b7cd7847901d4e53b87b2219819da6ba P ca5cccf002874a15b99a3ec1fdea4c3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 09:45:17.439854 17599 log.cc:1196] T b7cd7847901d4e53b87b2219819da6ba P ca5cccf002874a15b99a3ec1fdea4c3f: Deleting WAL directory at /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/minicluster-data/ts-1/wal/wals/b7cd7847901d4e53b87b2219819da6ba
I20260812 09:45:17.440166 17599 ts_tablet_manager.cc:1939] T b7cd7847901d4e53b87b2219819da6ba P ca5cccf002874a15b99a3ec1fdea4c3f: Deleting consensus metadata
I20260812 09:45:17.440538 17535 tablet_bootstrap.cc:492] T 3d64a70d4516472dacd20193ef9f01c0 P 9a386c33a01a49e4adf0dbcd90cf9223: Bootstrap starting.
I20260812 09:45:17.440645 17537 tablet_bootstrap.cc:492] T 3d64a70d4516472dacd20193ef9f01c0 P ca5cccf002874a15b99a3ec1fdea4c3f: Bootstrap starting.
I20260812 09:45:17.440673 17602 tablet_replica.cc:330] T b7cd7847901d4e53b87b2219819da6ba P dfe813ac860f4be68c2f87dc4bcab956: stopping tablet replica
I20260812 09:45:17.440966 17602 raft_consensus.cc:2229] T b7cd7847901d4e53b87b2219819da6ba P dfe813ac860f4be68c2f87dc4bcab956 [term 1 LEADER]: Raft consensus shutting down.
I20260812 09:45:17.441354 17602 raft_consensus.cc:2258] T b7cd7847901d4e53b87b2219819da6ba P dfe813ac860f4be68c2f87dc4bcab956 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 09:45:17.441385 17535 tablet_bootstrap.cc:654] T 3d64a70d4516472dacd20193ef9f01c0 P 9a386c33a01a49e4adf0dbcd90cf9223: Neither blocks nor log segments found. Creating new log.
I20260812 09:45:17.441471 17537 tablet_bootstrap.cc:654] T 3d64a70d4516472dacd20193ef9f01c0 P ca5cccf002874a15b99a3ec1fdea4c3f: Neither blocks nor log segments found. Creating new log.
I20260812 09:45:17.442173 16623 catalog_manager.cc:4841] TS 9a386c33a01a49e4adf0dbcd90cf9223 (127.13.99.1:42111): tablet b7cd7847901d4e53b87b2219819da6ba (table test-table [id=27de239c62a548a08ab67394ae56f125]) successfully deleted
I20260812 09:45:17.442709 17602 ts_tablet_manager.cc:1905] T b7cd7847901d4e53b87b2219819da6ba P dfe813ac860f4be68c2f87dc4bcab956: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 09:45:17.443250 16625 catalog_manager.cc:4841] TS ca5cccf002874a15b99a3ec1fdea4c3f (127.13.99.2:35267): tablet b7cd7847901d4e53b87b2219819da6ba (table test-table [id=27de239c62a548a08ab67394ae56f125]) successfully deleted
I20260812 09:45:17.443373 17382 tablet_service.cc:1467] Processing CreateTablet for tablet 3d64a70d4516472dacd20193ef9f01c0 (DEFAULT_TABLE table=test-table [id=f08f0287acab4770a6a7f00c8f9e6f79]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 09:45:17.443612 17382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d64a70d4516472dacd20193ef9f01c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:45:17.445538 17536 tablet_bootstrap.cc:492] T 3d64a70d4516472dacd20193ef9f01c0 P dfe813ac860f4be68c2f87dc4bcab956: Bootstrap starting.
I20260812 09:45:17.445892 17602 ts_tablet_manager.cc:1918] T b7cd7847901d4e53b87b2219819da6ba P dfe813ac860f4be68c2f87dc4bcab956: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 09:45:17.445994 17602 log.cc:1196] T b7cd7847901d4e53b87b2219819da6ba P dfe813ac860f4be68c2f87dc4bcab956: Deleting WAL directory at /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/minicluster-data/ts-2/wal/wals/b7cd7847901d4e53b87b2219819da6ba
I20260812 09:45:17.446529 17536 tablet_bootstrap.cc:654] T 3d64a70d4516472dacd20193ef9f01c0 P dfe813ac860f4be68c2f87dc4bcab956: Neither blocks nor log segments found. Creating new log.
I20260812 09:45:17.447110 17535 tablet_bootstrap.cc:492] T 3d64a70d4516472dacd20193ef9f01c0 P 9a386c33a01a49e4adf0dbcd90cf9223: No bootstrap required, opened a new log
I20260812 09:45:17.447185 17535 ts_tablet_manager.cc:1397] T 3d64a70d4516472dacd20193ef9f01c0 P 9a386c33a01a49e4adf0dbcd90cf9223: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260812 09:45:17.447656 17535 raft_consensus.cc:348] T 3d64a70d4516472dacd20193ef9f01c0 P 9a386c33a01a49e4adf0dbcd90cf9223 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a386c33a01a49e4adf0dbcd90cf9223" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 42111 } } peers { permanent_uuid: "ca5cccf002874a15b99a3ec1fdea4c3f" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35267 } } peers { permanent_uuid: "dfe813ac860f4be68c2f87dc4bcab956" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 35941 } }
I20260812 09:45:17.447795 17535 raft_consensus.cc:374] T 3d64a70d4516472dacd20193ef9f01c0 P 9a386c33a01a49e4adf0dbcd90cf9223 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:45:17.447835 17535 raft_consensus.cc:729] T 3d64a70d4516472dacd20193ef9f01c0 P 9a386c33a01a49e4adf0dbcd90cf9223 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a386c33a01a49e4adf0dbcd90cf9223, State: Initialized, Role: FOLLOWER
I20260812 09:45:17.448036 17535 consensus_queue.cc:260] T 3d64a70d4516472dacd20193ef9f01c0 P 9a386c33a01a49e4adf0dbcd90cf9223 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a386c33a01a49e4adf0dbcd90cf9223" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 42111 } } peers { permanent_uuid: "ca5cccf002874a15b99a3ec1fdea4c3f" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35267 } } peers { permanent_uuid: "dfe813ac860f4be68c2f87dc4bcab956" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 35941 } }
I20260812 09:45:17.448455 17536 tablet_bootstrap.cc:492] T 3d64a70d4516472dacd20193ef9f01c0 P dfe813ac860f4be68c2f87dc4bcab956: No bootstrap required, opened a new log
I20260812 09:45:17.448554 17536 ts_tablet_manager.cc:1397] T 3d64a70d4516472dacd20193ef9f01c0 P dfe813ac860f4be68c2f87dc4bcab956: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 09:45:17.448999 17536 raft_consensus.cc:348] T 3d64a70d4516472dacd20193ef9f01c0 P dfe813ac860f4be68c2f87dc4bcab956 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a386c33a01a49e4adf0dbcd90cf9223" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 42111 } } peers { permanent_uuid: "ca5cccf002874a15b99a3ec1fdea4c3f" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35267 } } peers { permanent_uuid: "dfe813ac860f4be68c2f87dc4bcab956" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 35941 } }
I20260812 09:45:17.449098 17536 raft_consensus.cc:374] T 3d64a70d4516472dacd20193ef9f01c0 P dfe813ac860f4be68c2f87dc4bcab956 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:45:17.449132 17536 raft_consensus.cc:729] T 3d64a70d4516472dacd20193ef9f01c0 P dfe813ac860f4be68c2f87dc4bcab956 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfe813ac860f4be68c2f87dc4bcab956, State: Initialized, Role: FOLLOWER
I20260812 09:45:17.449265 17536 consensus_queue.cc:260] T 3d64a70d4516472dacd20193ef9f01c0 P dfe813ac860f4be68c2f87dc4bcab956 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a386c33a01a49e4adf0dbcd90cf9223" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 42111 } } peers { permanent_uuid: "ca5cccf002874a15b99a3ec1fdea4c3f" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35267 } } peers { permanent_uuid: "dfe813ac860f4be68c2f87dc4bcab956" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 35941 } }
I20260812 09:45:17.449564 17536 ts_tablet_manager.cc:1428] T 3d64a70d4516472dacd20193ef9f01c0 P dfe813ac860f4be68c2f87dc4bcab956: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260812 09:45:17.450163 17535 ts_tablet_manager.cc:1428] T 3d64a70d4516472dacd20193ef9f01c0 P 9a386c33a01a49e4adf0dbcd90cf9223: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260812 09:45:17.451078 17602 ts_tablet_manager.cc:1939] T b7cd7847901d4e53b87b2219819da6ba P dfe813ac860f4be68c2f87dc4bcab956: Deleting consensus metadata
I20260812 09:45:17.452193 16621 catalog_manager.cc:4841] TS dfe813ac860f4be68c2f87dc4bcab956 (127.13.99.3:35941): tablet b7cd7847901d4e53b87b2219819da6ba (table test-table [id=27de239c62a548a08ab67394ae56f125]) successfully deleted
I20260812 09:45:17.453270 17537 tablet_bootstrap.cc:492] T 3d64a70d4516472dacd20193ef9f01c0 P ca5cccf002874a15b99a3ec1fdea4c3f: No bootstrap required, opened a new log
I20260812 09:45:17.453364 17537 ts_tablet_manager.cc:1397] T 3d64a70d4516472dacd20193ef9f01c0 P ca5cccf002874a15b99a3ec1fdea4c3f: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.000s
I20260812 09:45:17.453789 17537 raft_consensus.cc:348] T 3d64a70d4516472dacd20193ef9f01c0 P ca5cccf002874a15b99a3ec1fdea4c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a386c33a01a49e4adf0dbcd90cf9223" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 42111 } } peers { permanent_uuid: "ca5cccf002874a15b99a3ec1fdea4c3f" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35267 } } peers { permanent_uuid: "dfe813ac860f4be68c2f87dc4bcab956" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 35941 } }
I20260812 09:45:17.453899 17537 raft_consensus.cc:374] T 3d64a70d4516472dacd20193ef9f01c0 P ca5cccf002874a15b99a3ec1fdea4c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:45:17.453927 17537 raft_consensus.cc:729] T 3d64a70d4516472dacd20193ef9f01c0 P ca5cccf002874a15b99a3ec1fdea4c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca5cccf002874a15b99a3ec1fdea4c3f, State: Initialized, Role: FOLLOWER
I20260812 09:45:17.454052 17537 consensus_queue.cc:260] T 3d64a70d4516472dacd20193ef9f01c0 P ca5cccf002874a15b99a3ec1fdea4c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a386c33a01a49e4adf0dbcd90cf9223" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 42111 } } peers { permanent_uuid: "ca5cccf002874a15b99a3ec1fdea4c3f" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35267 } } peers { permanent_uuid: "dfe813ac860f4be68c2f87dc4bcab956" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 35941 } }
I20260812 09:45:17.454286 17537 ts_tablet_manager.cc:1428] T 3d64a70d4516472dacd20193ef9f01c0 P ca5cccf002874a15b99a3ec1fdea4c3f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260812 09:45:17.636709 17586 raft_consensus.cc:482] T 3d64a70d4516472dacd20193ef9f01c0 P dfe813ac860f4be68c2f87dc4bcab956 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 09:45:17.636891 17586 raft_consensus.cc:504] T 3d64a70d4516472dacd20193ef9f01c0 P dfe813ac860f4be68c2f87dc4bcab956 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a386c33a01a49e4adf0dbcd90cf9223" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 42111 } } peers { permanent_uuid: "ca5cccf002874a15b99a3ec1fdea4c3f" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35267 } } peers { permanent_uuid: "dfe813ac860f4be68c2f87dc4bcab956" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 35941 } }
I20260812 09:45:17.637382 17586 leader_election.cc:288] T 3d64a70d4516472dacd20193ef9f01c0 P dfe813ac860f4be68c2f87dc4bcab956 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a386c33a01a49e4adf0dbcd90cf9223 (127.13.99.1:42111), ca5cccf002874a15b99a3ec1fdea4c3f (127.13.99.2:35267)
I20260812 09:45:17.637854 16935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d64a70d4516472dacd20193ef9f01c0" candidate_uuid: "dfe813ac860f4be68c2f87dc4bcab956" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a386c33a01a49e4adf0dbcd90cf9223" is_pre_election: true
I20260812 09:45:17.637858 17170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d64a70d4516472dacd20193ef9f01c0" candidate_uuid: "dfe813ac860f4be68c2f87dc4bcab956" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca5cccf002874a15b99a3ec1fdea4c3f" is_pre_election: true
I20260812 09:45:17.638056 16935 raft_consensus.cc:2454] T 3d64a70d4516472dacd20193ef9f01c0 P 9a386c33a01a49e4adf0dbcd90cf9223 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfe813ac860f4be68c2f87dc4bcab956 in term 0.
I20260812 09:45:17.638056 17170 raft_consensus.cc:2454] T 3d64a70d4516472dacd20193ef9f01c0 P ca5cccf002874a15b99a3ec1fdea4c3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfe813ac860f4be68c2f87dc4bcab956 in term 0.
I20260812 09:45:17.638355 17304 leader_election.cc:302] T 3d64a70d4516472dacd20193ef9f01c0 P dfe813ac860f4be68c2f87dc4bcab956 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a386c33a01a49e4adf0dbcd90cf9223, dfe813ac860f4be68c2f87dc4bcab956; no voters:
I20260812 09:45:17.638535 17586 raft_consensus.cc:2781] T 3d64a70d4516472dacd20193ef9f01c0 P dfe813ac860f4be68c2f87dc4bcab956 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 09:45:17.638599 17586 raft_consensus.cc:482] T 3d64a70d4516472dacd20193ef9f01c0 P dfe813ac860f4be68c2f87dc4bcab956 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 09:45:17.638643 17586 raft_consensus.cc:3037] T 3d64a70d4516472dacd20193ef9f01c0 P dfe813ac860f4be68c2f87dc4bcab956 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:45:17.639667 17586 raft_consensus.cc:504] T 3d64a70d4516472dacd20193ef9f01c0 P dfe813ac860f4be68c2f87dc4bcab956 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a386c33a01a49e4adf0dbcd90cf9223" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 42111 } } peers { permanent_uuid: "ca5cccf002874a15b99a3ec1fdea4c3f" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35267 } } peers { permanent_uuid: "dfe813ac860f4be68c2f87dc4bcab956" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 35941 } }
I20260812 09:45:17.640050 17586 leader_election.cc:288] T 3d64a70d4516472dacd20193ef9f01c0 P dfe813ac860f4be68c2f87dc4bcab956 [CANDIDATE]: Term 1 election: Requested vote from peers 9a386c33a01a49e4adf0dbcd90cf9223 (127.13.99.1:42111), ca5cccf002874a15b99a3ec1fdea4c3f (127.13.99.2:35267)
I20260812 09:45:17.640396 16935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d64a70d4516472dacd20193ef9f01c0" candidate_uuid: "dfe813ac860f4be68c2f87dc4bcab956" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a386c33a01a49e4adf0dbcd90cf9223"
I20260812 09:45:17.640470 17170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d64a70d4516472dacd20193ef9f01c0" candidate_uuid: "dfe813ac860f4be68c2f87dc4bcab956" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca5cccf002874a15b99a3ec1fdea4c3f"
I20260812 09:45:17.640555 16935 raft_consensus.cc:3037] T 3d64a70d4516472dacd20193ef9f01c0 P 9a386c33a01a49e4adf0dbcd90cf9223 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:45:17.640642 17170 raft_consensus.cc:3037] T 3d64a70d4516472dacd20193ef9f01c0 P ca5cccf002874a15b99a3ec1fdea4c3f [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:45:17.641464 16935 raft_consensus.cc:2454] T 3d64a70d4516472dacd20193ef9f01c0 P 9a386c33a01a49e4adf0dbcd90cf9223 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfe813ac860f4be68c2f87dc4bcab956 in term 1.
I20260812 09:45:17.641621 17170 raft_consensus.cc:2454] T 3d64a70d4516472dacd20193ef9f01c0 P ca5cccf002874a15b99a3ec1fdea4c3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfe813ac860f4be68c2f87dc4bcab956 in term 1.
I20260812 09:45:17.641785 17304 leader_election.cc:302] T 3d64a70d4516472dacd20193ef9f01c0 P dfe813ac860f4be68c2f87dc4bcab956 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a386c33a01a49e4adf0dbcd90cf9223, dfe813ac860f4be68c2f87dc4bcab956; no voters:
I20260812 09:45:17.641990 17586 raft_consensus.cc:2781] T 3d64a70d4516472dacd20193ef9f01c0 P dfe813ac860f4be68c2f87dc4bcab956 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 09:45:17.642095 17586 raft_consensus.cc:686] T 3d64a70d4516472dacd20193ef9f01c0 P dfe813ac860f4be68c2f87dc4bcab956 [term 1 LEADER]: Becoming Leader. State: Replica: dfe813ac860f4be68c2f87dc4bcab956, State: Running, Role: LEADER
I20260812 09:45:17.642254 17586 consensus_queue.cc:237] T 3d64a70d4516472dacd20193ef9f01c0 P dfe813ac860f4be68c2f87dc4bcab956 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a386c33a01a49e4adf0dbcd90cf9223" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 42111 } } peers { permanent_uuid: "ca5cccf002874a15b99a3ec1fdea4c3f" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35267 } } peers { permanent_uuid: "dfe813ac860f4be68c2f87dc4bcab956" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 35941 } }
I20260812 09:45:17.643893 16643 catalog_manager.cc:5495] T 3d64a70d4516472dacd20193ef9f01c0 P dfe813ac860f4be68c2f87dc4bcab956 reported cstate change: term changed from 0 to 1, leader changed from <none> to dfe813ac860f4be68c2f87dc4bcab956 (127.13.99.3). New cstate: current_term: 1 leader_uuid: "dfe813ac860f4be68c2f87dc4bcab956" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a386c33a01a49e4adf0dbcd90cf9223" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 42111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca5cccf002874a15b99a3ec1fdea4c3f" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfe813ac860f4be68c2f87dc4bcab956" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 35941 } health_report { overall_health: HEALTHY } } }
I20260812 09:45:17.699584 17170 raft_consensus.cc:1261] T 3d64a70d4516472dacd20193ef9f01c0 P ca5cccf002874a15b99a3ec1fdea4c3f [term 1 FOLLOWER]: Refusing update from remote peer dfe813ac860f4be68c2f87dc4bcab956: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 09:45:17.700017 16935 raft_consensus.cc:1261] T 3d64a70d4516472dacd20193ef9f01c0 P 9a386c33a01a49e4adf0dbcd90cf9223 [term 1 FOLLOWER]: Refusing update from remote peer dfe813ac860f4be68c2f87dc4bcab956: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 09:45:17.700044 17553 consensus_queue.cc:1035] T 3d64a70d4516472dacd20193ef9f01c0 P dfe813ac860f4be68c2f87dc4bcab956 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5cccf002874a15b99a3ec1fdea4c3f" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 09:45:17.700431 17553 consensus_queue.cc:1035] T 3d64a70d4516472dacd20193ef9f01c0 P dfe813ac860f4be68c2f87dc4bcab956 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a386c33a01a49e4adf0dbcd90cf9223" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 42111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 09:45:17.714741 17578 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260812 09:45:19.267519 16643 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43112:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 09:45:19.267673 16643 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43112:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 09:45:19.270452 16643 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 77dec44a5cf34f3a9c0e1dfc5ff57cd4: Sending DeleteTablet for 3 replicas of tablet 3d64a70d4516472dacd20193ef9f01c0
I20260812 09:45:19.271347 17132 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d64a70d4516472dacd20193ef9f01c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 09:45:19 UTC) from {username='kudu', principal='kudu/127.13.99.62@KRBTEST.COM'} at 127.0.0.1:53428
I20260812 09:45:19.271319 17382 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d64a70d4516472dacd20193ef9f01c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 09:45:19 UTC) from {username='kudu', principal='kudu/127.13.99.62@KRBTEST.COM'} at 127.0.0.1:39042
I20260812 09:45:19.271363 16887 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d64a70d4516472dacd20193ef9f01c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 09:45:19 UTC) from {username='kudu', principal='kudu/127.13.99.62@KRBTEST.COM'} at 127.0.0.1:44768
I20260812 09:45:19.271754 17653 tablet_replica.cc:330] T 3d64a70d4516472dacd20193ef9f01c0 P ca5cccf002874a15b99a3ec1fdea4c3f: stopping tablet replica
I20260812 09:45:19.271865 17651 tablet_replica.cc:330] T 3d64a70d4516472dacd20193ef9f01c0 P 9a386c33a01a49e4adf0dbcd90cf9223: stopping tablet replica
I20260812 09:45:19.272003 17653 raft_consensus.cc:2229] T 3d64a70d4516472dacd20193ef9f01c0 P ca5cccf002874a15b99a3ec1fdea4c3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 09:45:19.272047 17652 tablet_replica.cc:330] T 3d64a70d4516472dacd20193ef9f01c0 P dfe813ac860f4be68c2f87dc4bcab956: stopping tablet replica
I20260812 09:45:19.272176 17652 raft_consensus.cc:2229] T 3d64a70d4516472dacd20193ef9f01c0 P dfe813ac860f4be68c2f87dc4bcab956 [term 1 LEADER]: Raft consensus shutting down.
I20260812 09:45:19.272203 17653 raft_consensus.cc:2258] T 3d64a70d4516472dacd20193ef9f01c0 P ca5cccf002874a15b99a3ec1fdea4c3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 09:45:19.272351 17651 raft_consensus.cc:2229] T 3d64a70d4516472dacd20193ef9f01c0 P 9a386c33a01a49e4adf0dbcd90cf9223 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 09:45:19.272663 17652 raft_consensus.cc:2258] T 3d64a70d4516472dacd20193ef9f01c0 P dfe813ac860f4be68c2f87dc4bcab956 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 09:45:19.272866 17651 raft_consensus.cc:2258] T 3d64a70d4516472dacd20193ef9f01c0 P 9a386c33a01a49e4adf0dbcd90cf9223 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 09:45:19.273000 17653 ts_tablet_manager.cc:1905] T 3d64a70d4516472dacd20193ef9f01c0 P ca5cccf002874a15b99a3ec1fdea4c3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 09:45:19.273615 17651 ts_tablet_manager.cc:1905] T 3d64a70d4516472dacd20193ef9f01c0 P 9a386c33a01a49e4adf0dbcd90cf9223: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 09:45:19.275727 17652 ts_tablet_manager.cc:1905] T 3d64a70d4516472dacd20193ef9f01c0 P dfe813ac860f4be68c2f87dc4bcab956: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 09:45:19.275870 17653 ts_tablet_manager.cc:1918] T 3d64a70d4516472dacd20193ef9f01c0 P ca5cccf002874a15b99a3ec1fdea4c3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 09:45:19.275998 17653 log.cc:1196] T 3d64a70d4516472dacd20193ef9f01c0 P ca5cccf002874a15b99a3ec1fdea4c3f: Deleting WAL directory at /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/minicluster-data/ts-1/wal/wals/3d64a70d4516472dacd20193ef9f01c0
I20260812 09:45:19.276293 17653 ts_tablet_manager.cc:1939] T 3d64a70d4516472dacd20193ef9f01c0 P ca5cccf002874a15b99a3ec1fdea4c3f: Deleting consensus metadata
I20260812 09:45:19.276769 17651 ts_tablet_manager.cc:1918] T 3d64a70d4516472dacd20193ef9f01c0 P 9a386c33a01a49e4adf0dbcd90cf9223: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 09:45:19.277081 17651 log.cc:1196] T 3d64a70d4516472dacd20193ef9f01c0 P 9a386c33a01a49e4adf0dbcd90cf9223: Deleting WAL directory at /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/minicluster-data/ts-0/wal/wals/3d64a70d4516472dacd20193ef9f01c0
I20260812 09:45:19.277498 17651 ts_tablet_manager.cc:1939] T 3d64a70d4516472dacd20193ef9f01c0 P 9a386c33a01a49e4adf0dbcd90cf9223: Deleting consensus metadata
I20260812 09:45:19.277479 16625 catalog_manager.cc:4841] TS ca5cccf002874a15b99a3ec1fdea4c3f (127.13.99.2:35267): tablet 3d64a70d4516472dacd20193ef9f01c0 (table test-table [id=f08f0287acab4770a6a7f00c8f9e6f79]) successfully deleted
I20260812 09:45:19.278203 16623 catalog_manager.cc:4841] TS 9a386c33a01a49e4adf0dbcd90cf9223 (127.13.99.1:42111): tablet 3d64a70d4516472dacd20193ef9f01c0 (table test-table [id=f08f0287acab4770a6a7f00c8f9e6f79]) successfully deleted
I20260812 09:45:19.279399 17652 ts_tablet_manager.cc:1918] T 3d64a70d4516472dacd20193ef9f01c0 P dfe813ac860f4be68c2f87dc4bcab956: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 09:45:19.279500 17652 log.cc:1196] T 3d64a70d4516472dacd20193ef9f01c0 P dfe813ac860f4be68c2f87dc4bcab956: Deleting WAL directory at /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/minicluster-data/ts-2/wal/wals/3d64a70d4516472dacd20193ef9f01c0
I20260812 09:45:19.279811 17652 ts_tablet_manager.cc:1939] T 3d64a70d4516472dacd20193ef9f01c0 P dfe813ac860f4be68c2f87dc4bcab956: Deleting consensus metadata
I20260812 09:45:19.280583 16621 catalog_manager.cc:4841] TS dfe813ac860f4be68c2f87dc4bcab956 (127.13.99.3:35941): tablet 3d64a70d4516472dacd20193ef9f01c0 (table test-table [id=f08f0287acab4770a6a7f00c8f9e6f79]) successfully deleted
Aug 12 09:45:19 dist-test-slave-jq4t krb5kdc[16579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527919, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 09:45:19 dist-test-slave-jq4t krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527919, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
I20260812 09:45:19.302047 17661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:19.293372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43120 (local address 127.13.99.62:37397)
0812 09:45:19.293644 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:19.293666 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:19.293762 (+ 96us) server_negotiation.cc:407] Connection header received
0812 09:45:19.293954 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:19.293958 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:19.294012 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:19.294108 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:19.294819 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:19.295388 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:19.296239 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:19.296527 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:19.296640 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:19.298930 (+ 2290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:19.298934 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:19.298939 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:19.298968 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:19.300292 (+ 1324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:19.300798 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:19.300802 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:19.300804 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:19.300893 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:19.301170 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:19.301174 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:19.301176 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:19.301352 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:19.301502 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:19.301748 (+ 246us) server_negotiation.cc:299] Negotiation successful
0812 09:45:19.301874 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":90,"threads_started":1}
I20260812 09:45:19.304528 16643 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43120:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 09:45:19.304997 16643 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 09:45:19.309711 16887 tablet_service.cc:1467] Processing CreateTablet for tablet 8cb39d1833c3429497b4921524a27dfb (DEFAULT_TABLE table=test-table [id=37267ca89c6846f2b60ef72b3a393b11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 09:45:19.309728 17132 tablet_service.cc:1467] Processing CreateTablet for tablet 8cb39d1833c3429497b4921524a27dfb (DEFAULT_TABLE table=test-table [id=37267ca89c6846f2b60ef72b3a393b11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 09:45:19.309800 17382 tablet_service.cc:1467] Processing CreateTablet for tablet 8cb39d1833c3429497b4921524a27dfb (DEFAULT_TABLE table=test-table [id=37267ca89c6846f2b60ef72b3a393b11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 09:45:19.310050 17132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cb39d1833c3429497b4921524a27dfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:45:19.310053 17382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cb39d1833c3429497b4921524a27dfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:45:19.310055 16887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cb39d1833c3429497b4921524a27dfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:45:19.312279 17662 tablet_bootstrap.cc:492] T 8cb39d1833c3429497b4921524a27dfb P ca5cccf002874a15b99a3ec1fdea4c3f: Bootstrap starting.
I20260812 09:45:19.313024 17663 tablet_bootstrap.cc:492] T 8cb39d1833c3429497b4921524a27dfb P 9a386c33a01a49e4adf0dbcd90cf9223: Bootstrap starting.
I20260812 09:45:19.313314 17662 tablet_bootstrap.cc:654] T 8cb39d1833c3429497b4921524a27dfb P ca5cccf002874a15b99a3ec1fdea4c3f: Neither blocks nor log segments found. Creating new log.
I20260812 09:45:19.313872 17663 tablet_bootstrap.cc:654] T 8cb39d1833c3429497b4921524a27dfb P 9a386c33a01a49e4adf0dbcd90cf9223: Neither blocks nor log segments found. Creating new log.
I20260812 09:45:19.314904 17664 tablet_bootstrap.cc:492] T 8cb39d1833c3429497b4921524a27dfb P dfe813ac860f4be68c2f87dc4bcab956: Bootstrap starting.
I20260812 09:45:19.314929 17662 tablet_bootstrap.cc:492] T 8cb39d1833c3429497b4921524a27dfb P ca5cccf002874a15b99a3ec1fdea4c3f: No bootstrap required, opened a new log
I20260812 09:45:19.315018 17662 ts_tablet_manager.cc:1397] T 8cb39d1833c3429497b4921524a27dfb P ca5cccf002874a15b99a3ec1fdea4c3f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 09:45:19.315550 17662 raft_consensus.cc:348] T 8cb39d1833c3429497b4921524a27dfb P ca5cccf002874a15b99a3ec1fdea4c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a386c33a01a49e4adf0dbcd90cf9223" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 42111 } } peers { permanent_uuid: "ca5cccf002874a15b99a3ec1fdea4c3f" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35267 } } peers { permanent_uuid: "dfe813ac860f4be68c2f87dc4bcab956" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 35941 } }
I20260812 09:45:19.315670 17662 raft_consensus.cc:374] T 8cb39d1833c3429497b4921524a27dfb P ca5cccf002874a15b99a3ec1fdea4c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:45:19.315714 17662 raft_consensus.cc:729] T 8cb39d1833c3429497b4921524a27dfb P ca5cccf002874a15b99a3ec1fdea4c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca5cccf002874a15b99a3ec1fdea4c3f, State: Initialized, Role: FOLLOWER
I20260812 09:45:19.315845 17664 tablet_bootstrap.cc:654] T 8cb39d1833c3429497b4921524a27dfb P dfe813ac860f4be68c2f87dc4bcab956: Neither blocks nor log segments found. Creating new log.
I20260812 09:45:19.315850 17662 consensus_queue.cc:260] T 8cb39d1833c3429497b4921524a27dfb P ca5cccf002874a15b99a3ec1fdea4c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a386c33a01a49e4adf0dbcd90cf9223" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 42111 } } peers { permanent_uuid: "ca5cccf002874a15b99a3ec1fdea4c3f" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35267 } } peers { permanent_uuid: "dfe813ac860f4be68c2f87dc4bcab956" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 35941 } }
I20260812 09:45:19.316242 17662 ts_tablet_manager.cc:1428] T 8cb39d1833c3429497b4921524a27dfb P ca5cccf002874a15b99a3ec1fdea4c3f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 09:45:19.316916 17663 tablet_bootstrap.cc:492] T 8cb39d1833c3429497b4921524a27dfb P 9a386c33a01a49e4adf0dbcd90cf9223: No bootstrap required, opened a new log
I20260812 09:45:19.317011 17663 ts_tablet_manager.cc:1397] T 8cb39d1833c3429497b4921524a27dfb P 9a386c33a01a49e4adf0dbcd90cf9223: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260812 09:45:19.317147 17664 tablet_bootstrap.cc:492] T 8cb39d1833c3429497b4921524a27dfb P dfe813ac860f4be68c2f87dc4bcab956: No bootstrap required, opened a new log
I20260812 09:45:19.317230 17664 ts_tablet_manager.cc:1397] T 8cb39d1833c3429497b4921524a27dfb P dfe813ac860f4be68c2f87dc4bcab956: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 09:45:19.317463 17663 raft_consensus.cc:348] T 8cb39d1833c3429497b4921524a27dfb P 9a386c33a01a49e4adf0dbcd90cf9223 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a386c33a01a49e4adf0dbcd90cf9223" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 42111 } } peers { permanent_uuid: "ca5cccf002874a15b99a3ec1fdea4c3f" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35267 } } peers { permanent_uuid: "dfe813ac860f4be68c2f87dc4bcab956" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 35941 } }
I20260812 09:45:19.317646 17663 raft_consensus.cc:374] T 8cb39d1833c3429497b4921524a27dfb P 9a386c33a01a49e4adf0dbcd90cf9223 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:45:19.317631 17664 raft_consensus.cc:348] T 8cb39d1833c3429497b4921524a27dfb P dfe813ac860f4be68c2f87dc4bcab956 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a386c33a01a49e4adf0dbcd90cf9223" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 42111 } } peers { permanent_uuid: "ca5cccf002874a15b99a3ec1fdea4c3f" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35267 } } peers { permanent_uuid: "dfe813ac860f4be68c2f87dc4bcab956" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 35941 } }
I20260812 09:45:19.317728 17663 raft_consensus.cc:729] T 8cb39d1833c3429497b4921524a27dfb P 9a386c33a01a49e4adf0dbcd90cf9223 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a386c33a01a49e4adf0dbcd90cf9223, State: Initialized, Role: FOLLOWER
I20260812 09:45:19.317785 17664 raft_consensus.cc:374] T 8cb39d1833c3429497b4921524a27dfb P dfe813ac860f4be68c2f87dc4bcab956 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:45:19.317826 17664 raft_consensus.cc:729] T 8cb39d1833c3429497b4921524a27dfb P dfe813ac860f4be68c2f87dc4bcab956 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfe813ac860f4be68c2f87dc4bcab956, State: Initialized, Role: FOLLOWER
I20260812 09:45:19.317961 17663 consensus_queue.cc:260] T 8cb39d1833c3429497b4921524a27dfb P 9a386c33a01a49e4adf0dbcd90cf9223 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a386c33a01a49e4adf0dbcd90cf9223" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 42111 } } peers { permanent_uuid: "ca5cccf002874a15b99a3ec1fdea4c3f" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35267 } } peers { permanent_uuid: "dfe813ac860f4be68c2f87dc4bcab956" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 35941 } }
I20260812 09:45:19.317983 17664 consensus_queue.cc:260] T 8cb39d1833c3429497b4921524a27dfb P dfe813ac860f4be68c2f87dc4bcab956 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a386c33a01a49e4adf0dbcd90cf9223" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 42111 } } peers { permanent_uuid: "ca5cccf002874a15b99a3ec1fdea4c3f" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35267 } } peers { permanent_uuid: "dfe813ac860f4be68c2f87dc4bcab956" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 35941 } }
I20260812 09:45:19.318267 17664 ts_tablet_manager.cc:1428] T 8cb39d1833c3429497b4921524a27dfb P dfe813ac860f4be68c2f87dc4bcab956: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 09:45:19.318367 17663 ts_tablet_manager.cc:1428] T 8cb39d1833c3429497b4921524a27dfb P 9a386c33a01a49e4adf0dbcd90cf9223: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260812 09:45:19.362438 17619 raft_consensus.cc:482] T 8cb39d1833c3429497b4921524a27dfb P dfe813ac860f4be68c2f87dc4bcab956 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 09:45:19.362593 17619 raft_consensus.cc:504] T 8cb39d1833c3429497b4921524a27dfb P dfe813ac860f4be68c2f87dc4bcab956 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a386c33a01a49e4adf0dbcd90cf9223" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 42111 } } peers { permanent_uuid: "ca5cccf002874a15b99a3ec1fdea4c3f" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35267 } } peers { permanent_uuid: "dfe813ac860f4be68c2f87dc4bcab956" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 35941 } }
I20260812 09:45:19.363137 17619 leader_election.cc:288] T 8cb39d1833c3429497b4921524a27dfb P dfe813ac860f4be68c2f87dc4bcab956 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a386c33a01a49e4adf0dbcd90cf9223 (127.13.99.1:42111), ca5cccf002874a15b99a3ec1fdea4c3f (127.13.99.2:35267)
I20260812 09:45:19.363595 16927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cb39d1833c3429497b4921524a27dfb" candidate_uuid: "dfe813ac860f4be68c2f87dc4bcab956" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a386c33a01a49e4adf0dbcd90cf9223" is_pre_election: true
I20260812 09:45:19.363610 17170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cb39d1833c3429497b4921524a27dfb" candidate_uuid: "dfe813ac860f4be68c2f87dc4bcab956" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca5cccf002874a15b99a3ec1fdea4c3f" is_pre_election: true
I20260812 09:45:19.363775 17170 raft_consensus.cc:2454] T 8cb39d1833c3429497b4921524a27dfb P ca5cccf002874a15b99a3ec1fdea4c3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfe813ac860f4be68c2f87dc4bcab956 in term 0.
I20260812 09:45:19.363768 16927 raft_consensus.cc:2454] T 8cb39d1833c3429497b4921524a27dfb P 9a386c33a01a49e4adf0dbcd90cf9223 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfe813ac860f4be68c2f87dc4bcab956 in term 0.
I20260812 09:45:19.364136 17307 leader_election.cc:302] T 8cb39d1833c3429497b4921524a27dfb P dfe813ac860f4be68c2f87dc4bcab956 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ca5cccf002874a15b99a3ec1fdea4c3f, dfe813ac860f4be68c2f87dc4bcab956; no voters:
I20260812 09:45:19.364321 17619 raft_consensus.cc:2781] T 8cb39d1833c3429497b4921524a27dfb P dfe813ac860f4be68c2f87dc4bcab956 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 09:45:19.364387 17619 raft_consensus.cc:482] T 8cb39d1833c3429497b4921524a27dfb P dfe813ac860f4be68c2f87dc4bcab956 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 09:45:19.364425 17619 raft_consensus.cc:3037] T 8cb39d1833c3429497b4921524a27dfb P dfe813ac860f4be68c2f87dc4bcab956 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:45:19.365466 17619 raft_consensus.cc:504] T 8cb39d1833c3429497b4921524a27dfb P dfe813ac860f4be68c2f87dc4bcab956 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a386c33a01a49e4adf0dbcd90cf9223" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 42111 } } peers { permanent_uuid: "ca5cccf002874a15b99a3ec1fdea4c3f" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35267 } } peers { permanent_uuid: "dfe813ac860f4be68c2f87dc4bcab956" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 35941 } }
I20260812 09:45:19.365778 17619 leader_election.cc:288] T 8cb39d1833c3429497b4921524a27dfb P dfe813ac860f4be68c2f87dc4bcab956 [CANDIDATE]: Term 1 election: Requested vote from peers 9a386c33a01a49e4adf0dbcd90cf9223 (127.13.99.1:42111), ca5cccf002874a15b99a3ec1fdea4c3f (127.13.99.2:35267)
I20260812 09:45:19.366132 16927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cb39d1833c3429497b4921524a27dfb" candidate_uuid: "dfe813ac860f4be68c2f87dc4bcab956" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a386c33a01a49e4adf0dbcd90cf9223"
I20260812 09:45:19.366132 17170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cb39d1833c3429497b4921524a27dfb" candidate_uuid: "dfe813ac860f4be68c2f87dc4bcab956" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca5cccf002874a15b99a3ec1fdea4c3f"
I20260812 09:45:19.366252 17170 raft_consensus.cc:3037] T 8cb39d1833c3429497b4921524a27dfb P ca5cccf002874a15b99a3ec1fdea4c3f [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:45:19.366277 16927 raft_consensus.cc:3037] T 8cb39d1833c3429497b4921524a27dfb P 9a386c33a01a49e4adf0dbcd90cf9223 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:45:19.367414 16927 raft_consensus.cc:2454] T 8cb39d1833c3429497b4921524a27dfb P 9a386c33a01a49e4adf0dbcd90cf9223 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfe813ac860f4be68c2f87dc4bcab956 in term 1.
I20260812 09:45:19.367405 17170 raft_consensus.cc:2454] T 8cb39d1833c3429497b4921524a27dfb P ca5cccf002874a15b99a3ec1fdea4c3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfe813ac860f4be68c2f87dc4bcab956 in term 1.
I20260812 09:45:19.367695 17307 leader_election.cc:302] T 8cb39d1833c3429497b4921524a27dfb P dfe813ac860f4be68c2f87dc4bcab956 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ca5cccf002874a15b99a3ec1fdea4c3f, dfe813ac860f4be68c2f87dc4bcab956; no voters:
I20260812 09:45:19.367865 17619 raft_consensus.cc:2781] T 8cb39d1833c3429497b4921524a27dfb P dfe813ac860f4be68c2f87dc4bcab956 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 09:45:19.367965 17619 raft_consensus.cc:686] T 8cb39d1833c3429497b4921524a27dfb P dfe813ac860f4be68c2f87dc4bcab956 [term 1 LEADER]: Becoming Leader. State: Replica: dfe813ac860f4be68c2f87dc4bcab956, State: Running, Role: LEADER
I20260812 09:45:19.368108 17619 consensus_queue.cc:237] T 8cb39d1833c3429497b4921524a27dfb P dfe813ac860f4be68c2f87dc4bcab956 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a386c33a01a49e4adf0dbcd90cf9223" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 42111 } } peers { permanent_uuid: "ca5cccf002874a15b99a3ec1fdea4c3f" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35267 } } peers { permanent_uuid: "dfe813ac860f4be68c2f87dc4bcab956" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 35941 } }
I20260812 09:45:19.369827 16641 catalog_manager.cc:5495] T 8cb39d1833c3429497b4921524a27dfb P dfe813ac860f4be68c2f87dc4bcab956 reported cstate change: term changed from 0 to 1, leader changed from <none> to dfe813ac860f4be68c2f87dc4bcab956 (127.13.99.3). New cstate: current_term: 1 leader_uuid: "dfe813ac860f4be68c2f87dc4bcab956" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a386c33a01a49e4adf0dbcd90cf9223" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 42111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca5cccf002874a15b99a3ec1fdea4c3f" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfe813ac860f4be68c2f87dc4bcab956" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 35941 } health_report { overall_health: HEALTHY } } }
I20260812 09:45:19.389307 17678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:19.385941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39070 (local address 127.13.99.3:35941)
0812 09:45:19.386206 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:19.386215 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:19.386231 (+ 16us) server_negotiation.cc:407] Connection header received
0812 09:45:19.386355 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:19.386358 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:19.386415 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:19.386519 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 09:45:19.386812 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:19.387330 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:19.388175 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:19.388423 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:19.388606 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:19.388830 (+ 224us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 09:45:19.388906 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 09:45:19.388962 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:19.389068 (+ 106us) server_negotiation.cc:299] Negotiation successful
0812 09:45:19.389114 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":122,"threads_started":1}
I20260812 09:45:19.395932 16927 raft_consensus.cc:1261] T 8cb39d1833c3429497b4921524a27dfb P 9a386c33a01a49e4adf0dbcd90cf9223 [term 1 FOLLOWER]: Refusing update from remote peer dfe813ac860f4be68c2f87dc4bcab956: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 09:45:19.396330 17170 raft_consensus.cc:1261] T 8cb39d1833c3429497b4921524a27dfb P ca5cccf002874a15b99a3ec1fdea4c3f [term 1 FOLLOWER]: Refusing update from remote peer dfe813ac860f4be68c2f87dc4bcab956: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 09:45:19.396380 17627 consensus_queue.cc:1035] T 8cb39d1833c3429497b4921524a27dfb P dfe813ac860f4be68c2f87dc4bcab956 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a386c33a01a49e4adf0dbcd90cf9223" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 42111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 09:45:19.396749 17619 consensus_queue.cc:1035] T 8cb39d1833c3429497b4921524a27dfb P dfe813ac860f4be68c2f87dc4bcab956 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5cccf002874a15b99a3ec1fdea4c3f" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 09:45:19.429337 16645 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43120:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 09:45:19.429519 16645 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43120:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 09:45:19.431738 16645 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 77dec44a5cf34f3a9c0e1dfc5ff57cd4: Sending DeleteTablet for 3 replicas of tablet 8cb39d1833c3429497b4921524a27dfb
I20260812 09:45:19.432327 17382 tablet_service.cc:1514] Processing DeleteTablet for tablet 8cb39d1833c3429497b4921524a27dfb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 09:45:19 UTC) from {username='kudu', principal='kudu/127.13.99.62@KRBTEST.COM'} at 127.0.0.1:39042
I20260812 09:45:19.432327 17132 tablet_service.cc:1514] Processing DeleteTablet for tablet 8cb39d1833c3429497b4921524a27dfb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 09:45:19 UTC) from {username='kudu', principal='kudu/127.13.99.62@KRBTEST.COM'} at 127.0.0.1:53428
I20260812 09:45:19.432555 16887 tablet_service.cc:1514] Processing DeleteTablet for tablet 8cb39d1833c3429497b4921524a27dfb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 09:45:19 UTC) from {username='kudu', principal='kudu/127.13.99.62@KRBTEST.COM'} at 127.0.0.1:44768
I20260812 09:45:19.432631 17652 tablet_replica.cc:330] T 8cb39d1833c3429497b4921524a27dfb P dfe813ac860f4be68c2f87dc4bcab956: stopping tablet replica
I20260812 09:45:19.432791 17653 tablet_replica.cc:330] T 8cb39d1833c3429497b4921524a27dfb P ca5cccf002874a15b99a3ec1fdea4c3f: stopping tablet replica
I20260812 09:45:19.432782 17652 raft_consensus.cc:2229] T 8cb39d1833c3429497b4921524a27dfb P dfe813ac860f4be68c2f87dc4bcab956 [term 1 LEADER]: Raft consensus shutting down.
I20260812 09:45:19.432935 17653 raft_consensus.cc:2229] T 8cb39d1833c3429497b4921524a27dfb P ca5cccf002874a15b99a3ec1fdea4c3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 09:45:19.432965 17652 raft_consensus.cc:2258] T 8cb39d1833c3429497b4921524a27dfb P dfe813ac860f4be68c2f87dc4bcab956 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 09:45:19.433081 17653 raft_consensus.cc:2258] T 8cb39d1833c3429497b4921524a27dfb P ca5cccf002874a15b99a3ec1fdea4c3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 09:45:19.433274 17651 tablet_replica.cc:330] T 8cb39d1833c3429497b4921524a27dfb P 9a386c33a01a49e4adf0dbcd90cf9223: stopping tablet replica
I20260812 09:45:19.433403 17651 raft_consensus.cc:2229] T 8cb39d1833c3429497b4921524a27dfb P 9a386c33a01a49e4adf0dbcd90cf9223 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 09:45:19.433523 17651 raft_consensus.cc:2258] T 8cb39d1833c3429497b4921524a27dfb P 9a386c33a01a49e4adf0dbcd90cf9223 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 09:45:19.433699 16645 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43120:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260812 09:45:19.433933 17653 ts_tablet_manager.cc:1905] T 8cb39d1833c3429497b4921524a27dfb P ca5cccf002874a15b99a3ec1fdea4c3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 09:45:19.434538 17651 ts_tablet_manager.cc:1905] T 8cb39d1833c3429497b4921524a27dfb P 9a386c33a01a49e4adf0dbcd90cf9223: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 09:45:19.434546 17652 ts_tablet_manager.cc:1905] T 8cb39d1833c3429497b4921524a27dfb P dfe813ac860f4be68c2f87dc4bcab956: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 09:45:19.436971 17653 ts_tablet_manager.cc:1918] T 8cb39d1833c3429497b4921524a27dfb P ca5cccf002874a15b99a3ec1fdea4c3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 09:45:19.437119 17653 log.cc:1196] T 8cb39d1833c3429497b4921524a27dfb P ca5cccf002874a15b99a3ec1fdea4c3f: Deleting WAL directory at /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/minicluster-data/ts-1/wal/wals/8cb39d1833c3429497b4921524a27dfb
I20260812 09:45:19.437189 17651 ts_tablet_manager.cc:1918] T 8cb39d1833c3429497b4921524a27dfb P 9a386c33a01a49e4adf0dbcd90cf9223: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 09:45:19.437263 17651 log.cc:1196] T 8cb39d1833c3429497b4921524a27dfb P 9a386c33a01a49e4adf0dbcd90cf9223: Deleting WAL directory at /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/minicluster-data/ts-0/wal/wals/8cb39d1833c3429497b4921524a27dfb
I20260812 09:45:19.437476 17653 ts_tablet_manager.cc:1939] T 8cb39d1833c3429497b4921524a27dfb P ca5cccf002874a15b99a3ec1fdea4c3f: Deleting consensus metadata
I20260812 09:45:19.437515 17651 ts_tablet_manager.cc:1939] T 8cb39d1833c3429497b4921524a27dfb P 9a386c33a01a49e4adf0dbcd90cf9223: Deleting consensus metadata
I20260812 09:45:19.437564 17652 ts_tablet_manager.cc:1918] T 8cb39d1833c3429497b4921524a27dfb P dfe813ac860f4be68c2f87dc4bcab956: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 09:45:19.437628 17652 log.cc:1196] T 8cb39d1833c3429497b4921524a27dfb P dfe813ac860f4be68c2f87dc4bcab956: Deleting WAL directory at /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/minicluster-data/ts-2/wal/wals/8cb39d1833c3429497b4921524a27dfb
I20260812 09:45:19.437872 17652 ts_tablet_manager.cc:1939] T 8cb39d1833c3429497b4921524a27dfb P dfe813ac860f4be68c2f87dc4bcab956: Deleting consensus metadata
I20260812 09:45:19.438202 16623 catalog_manager.cc:4841] TS 9a386c33a01a49e4adf0dbcd90cf9223 (127.13.99.1:42111): tablet 8cb39d1833c3429497b4921524a27dfb (table test-table [id=37267ca89c6846f2b60ef72b3a393b11]) successfully deleted
I20260812 09:45:19.438221 16625 catalog_manager.cc:4841] TS ca5cccf002874a15b99a3ec1fdea4c3f (127.13.99.2:35267): tablet 8cb39d1833c3429497b4921524a27dfb (table test-table [id=37267ca89c6846f2b60ef72b3a393b11]) successfully deleted
I20260812 09:45:19.438414 16621 catalog_manager.cc:4841] TS dfe813ac860f4be68c2f87dc4bcab956 (127.13.99.3:35941): tablet 8cb39d1833c3429497b4921524a27dfb (table test-table [id=37267ca89c6846f2b60ef72b3a393b11]) successfully deleted
W20260812 09:45:19.438486 16645 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 09:45:19.443013 16887 tablet_service.cc:1467] Processing CreateTablet for tablet 343ecbaeb4e246b892cd377f29eb0919 (DEFAULT_TABLE table=test-table [id=12f1a5c42ac34faaa1a5f1b4336005d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 09:45:19.443223 17382 tablet_service.cc:1467] Processing CreateTablet for tablet 343ecbaeb4e246b892cd377f29eb0919 (DEFAULT_TABLE table=test-table [id=12f1a5c42ac34faaa1a5f1b4336005d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 09:45:19.443282 16887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 343ecbaeb4e246b892cd377f29eb0919. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:45:19.443457 17382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 343ecbaeb4e246b892cd377f29eb0919. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:45:19.444767 17132 tablet_service.cc:1467] Processing CreateTablet for tablet 343ecbaeb4e246b892cd377f29eb0919 (DEFAULT_TABLE table=test-table [id=12f1a5c42ac34faaa1a5f1b4336005d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 09:45:19.444972 17132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 343ecbaeb4e246b892cd377f29eb0919. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:45:19.445994 17664 tablet_bootstrap.cc:492] T 343ecbaeb4e246b892cd377f29eb0919 P dfe813ac860f4be68c2f87dc4bcab956: Bootstrap starting.
I20260812 09:45:19.446838 17664 tablet_bootstrap.cc:654] T 343ecbaeb4e246b892cd377f29eb0919 P dfe813ac860f4be68c2f87dc4bcab956: Neither blocks nor log segments found. Creating new log.
I20260812 09:45:19.447196 17662 tablet_bootstrap.cc:492] T 343ecbaeb4e246b892cd377f29eb0919 P ca5cccf002874a15b99a3ec1fdea4c3f: Bootstrap starting.
I20260812 09:45:19.447263 17663 tablet_bootstrap.cc:492] T 343ecbaeb4e246b892cd377f29eb0919 P 9a386c33a01a49e4adf0dbcd90cf9223: Bootstrap starting.
I20260812 09:45:19.447981 17664 tablet_bootstrap.cc:492] T 343ecbaeb4e246b892cd377f29eb0919 P dfe813ac860f4be68c2f87dc4bcab956: No bootstrap required, opened a new log
I20260812 09:45:19.448091 17664 ts_tablet_manager.cc:1397] T 343ecbaeb4e246b892cd377f29eb0919 P dfe813ac860f4be68c2f87dc4bcab956: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 09:45:19.448127 17662 tablet_bootstrap.cc:654] T 343ecbaeb4e246b892cd377f29eb0919 P ca5cccf002874a15b99a3ec1fdea4c3f: Neither blocks nor log segments found. Creating new log.
I20260812 09:45:19.448151 17663 tablet_bootstrap.cc:654] T 343ecbaeb4e246b892cd377f29eb0919 P 9a386c33a01a49e4adf0dbcd90cf9223: Neither blocks nor log segments found. Creating new log.
I20260812 09:45:19.448547 17664 raft_consensus.cc:348] T 343ecbaeb4e246b892cd377f29eb0919 P dfe813ac860f4be68c2f87dc4bcab956 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a386c33a01a49e4adf0dbcd90cf9223" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 42111 } } peers { permanent_uuid: "ca5cccf002874a15b99a3ec1fdea4c3f" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35267 } } peers { permanent_uuid: "dfe813ac860f4be68c2f87dc4bcab956" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 35941 } }
I20260812 09:45:19.448673 17664 raft_consensus.cc:374] T 343ecbaeb4e246b892cd377f29eb0919 P dfe813ac860f4be68c2f87dc4bcab956 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:45:19.448720 17664 raft_consensus.cc:729] T 343ecbaeb4e246b892cd377f29eb0919 P dfe813ac860f4be68c2f87dc4bcab956 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfe813ac860f4be68c2f87dc4bcab956, State: Initialized, Role: FOLLOWER
I20260812 09:45:19.448845 17664 consensus_queue.cc:260] T 343ecbaeb4e246b892cd377f29eb0919 P dfe813ac860f4be68c2f87dc4bcab956 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a386c33a01a49e4adf0dbcd90cf9223" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 42111 } } peers { permanent_uuid: "ca5cccf002874a15b99a3ec1fdea4c3f" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35267 } } peers { permanent_uuid: "dfe813ac860f4be68c2f87dc4bcab956" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 35941 } }
I20260812 09:45:19.449124 17664 ts_tablet_manager.cc:1428] T 343ecbaeb4e246b892cd377f29eb0919 P dfe813ac860f4be68c2f87dc4bcab956: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 09:45:19.449443 17663 tablet_bootstrap.cc:492] T 343ecbaeb4e246b892cd377f29eb0919 P 9a386c33a01a49e4adf0dbcd90cf9223: No bootstrap required, opened a new log
I20260812 09:45:19.449539 17663 ts_tablet_manager.cc:1397] T 343ecbaeb4e246b892cd377f29eb0919 P 9a386c33a01a49e4adf0dbcd90cf9223: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260812 09:45:19.449815 17662 tablet_bootstrap.cc:492] T 343ecbaeb4e246b892cd377f29eb0919 P ca5cccf002874a15b99a3ec1fdea4c3f: No bootstrap required, opened a new log
I20260812 09:45:19.449899 17662 ts_tablet_manager.cc:1397] T 343ecbaeb4e246b892cd377f29eb0919 P ca5cccf002874a15b99a3ec1fdea4c3f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 09:45:19.449951 17663 raft_consensus.cc:348] T 343ecbaeb4e246b892cd377f29eb0919 P 9a386c33a01a49e4adf0dbcd90cf9223 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a386c33a01a49e4adf0dbcd90cf9223" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 42111 } } peers { permanent_uuid: "ca5cccf002874a15b99a3ec1fdea4c3f" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35267 } } peers { permanent_uuid: "dfe813ac860f4be68c2f87dc4bcab956" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 35941 } }
I20260812 09:45:19.450047 17663 raft_consensus.cc:374] T 343ecbaeb4e246b892cd377f29eb0919 P 9a386c33a01a49e4adf0dbcd90cf9223 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:45:19.450068 17663 raft_consensus.cc:729] T 343ecbaeb4e246b892cd377f29eb0919 P 9a386c33a01a49e4adf0dbcd90cf9223 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a386c33a01a49e4adf0dbcd90cf9223, State: Initialized, Role: FOLLOWER
I20260812 09:45:19.450155 17663 consensus_queue.cc:260] T 343ecbaeb4e246b892cd377f29eb0919 P 9a386c33a01a49e4adf0dbcd90cf9223 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a386c33a01a49e4adf0dbcd90cf9223" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 42111 } } peers { permanent_uuid: "ca5cccf002874a15b99a3ec1fdea4c3f" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35267 } } peers { permanent_uuid: "dfe813ac860f4be68c2f87dc4bcab956" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 35941 } }
I20260812 09:45:19.450398 17663 ts_tablet_manager.cc:1428] T 343ecbaeb4e246b892cd377f29eb0919 P 9a386c33a01a49e4adf0dbcd90cf9223: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260812 09:45:19.450372 17662 raft_consensus.cc:348] T 343ecbaeb4e246b892cd377f29eb0919 P ca5cccf002874a15b99a3ec1fdea4c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a386c33a01a49e4adf0dbcd90cf9223" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 42111 } } peers { permanent_uuid: "ca5cccf002874a15b99a3ec1fdea4c3f" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35267 } } peers { permanent_uuid: "dfe813ac860f4be68c2f87dc4bcab956" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 35941 } }
I20260812 09:45:19.450475 17662 raft_consensus.cc:374] T 343ecbaeb4e246b892cd377f29eb0919 P ca5cccf002874a15b99a3ec1fdea4c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:45:19.450515 17662 raft_consensus.cc:729] T 343ecbaeb4e246b892cd377f29eb0919 P ca5cccf002874a15b99a3ec1fdea4c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca5cccf002874a15b99a3ec1fdea4c3f, State: Initialized, Role: FOLLOWER
I20260812 09:45:19.450659 17662 consensus_queue.cc:260] T 343ecbaeb4e246b892cd377f29eb0919 P ca5cccf002874a15b99a3ec1fdea4c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a386c33a01a49e4adf0dbcd90cf9223" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 42111 } } peers { permanent_uuid: "ca5cccf002874a15b99a3ec1fdea4c3f" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35267 } } peers { permanent_uuid: "dfe813ac860f4be68c2f87dc4bcab956" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 35941 } }
I20260812 09:45:19.450886 17662 ts_tablet_manager.cc:1428] T 343ecbaeb4e246b892cd377f29eb0919 P ca5cccf002874a15b99a3ec1fdea4c3f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260812 09:45:19.494136 17616 raft_consensus.cc:482] T 343ecbaeb4e246b892cd377f29eb0919 P dfe813ac860f4be68c2f87dc4bcab956 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 09:45:19.494282 17616 raft_consensus.cc:504] T 343ecbaeb4e246b892cd377f29eb0919 P dfe813ac860f4be68c2f87dc4bcab956 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a386c33a01a49e4adf0dbcd90cf9223" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 42111 } } peers { permanent_uuid: "ca5cccf002874a15b99a3ec1fdea4c3f" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35267 } } peers { permanent_uuid: "dfe813ac860f4be68c2f87dc4bcab956" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 35941 } }
I20260812 09:45:19.494704 17616 leader_election.cc:288] T 343ecbaeb4e246b892cd377f29eb0919 P dfe813ac860f4be68c2f87dc4bcab956 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a386c33a01a49e4adf0dbcd90cf9223 (127.13.99.1:42111), ca5cccf002874a15b99a3ec1fdea4c3f (127.13.99.2:35267)
I20260812 09:45:19.495200 16927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "343ecbaeb4e246b892cd377f29eb0919" candidate_uuid: "dfe813ac860f4be68c2f87dc4bcab956" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a386c33a01a49e4adf0dbcd90cf9223" is_pre_election: true
I20260812 09:45:19.495219 17169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "343ecbaeb4e246b892cd377f29eb0919" candidate_uuid: "dfe813ac860f4be68c2f87dc4bcab956" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca5cccf002874a15b99a3ec1fdea4c3f" is_pre_election: true
I20260812 09:45:19.495369 17169 raft_consensus.cc:2454] T 343ecbaeb4e246b892cd377f29eb0919 P ca5cccf002874a15b99a3ec1fdea4c3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfe813ac860f4be68c2f87dc4bcab956 in term 0.
I20260812 09:45:19.495369 16927 raft_consensus.cc:2454] T 343ecbaeb4e246b892cd377f29eb0919 P 9a386c33a01a49e4adf0dbcd90cf9223 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfe813ac860f4be68c2f87dc4bcab956 in term 0.
I20260812 09:45:19.495697 17307 leader_election.cc:302] T 343ecbaeb4e246b892cd377f29eb0919 P dfe813ac860f4be68c2f87dc4bcab956 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ca5cccf002874a15b99a3ec1fdea4c3f, dfe813ac860f4be68c2f87dc4bcab956; no voters:
I20260812 09:45:19.495908 17616 raft_consensus.cc:2781] T 343ecbaeb4e246b892cd377f29eb0919 P dfe813ac860f4be68c2f87dc4bcab956 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 09:45:19.495968 17616 raft_consensus.cc:482] T 343ecbaeb4e246b892cd377f29eb0919 P dfe813ac860f4be68c2f87dc4bcab956 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 09:45:19.496027 17616 raft_consensus.cc:3037] T 343ecbaeb4e246b892cd377f29eb0919 P dfe813ac860f4be68c2f87dc4bcab956 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:45:19.497080 17616 raft_consensus.cc:504] T 343ecbaeb4e246b892cd377f29eb0919 P dfe813ac860f4be68c2f87dc4bcab956 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a386c33a01a49e4adf0dbcd90cf9223" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 42111 } } peers { permanent_uuid: "ca5cccf002874a15b99a3ec1fdea4c3f" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35267 } } peers { permanent_uuid: "dfe813ac860f4be68c2f87dc4bcab956" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 35941 } }
I20260812 09:45:19.497409 17616 leader_election.cc:288] T 343ecbaeb4e246b892cd377f29eb0919 P dfe813ac860f4be68c2f87dc4bcab956 [CANDIDATE]: Term 1 election: Requested vote from peers 9a386c33a01a49e4adf0dbcd90cf9223 (127.13.99.1:42111), ca5cccf002874a15b99a3ec1fdea4c3f (127.13.99.2:35267)
I20260812 09:45:19.497741 17169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "343ecbaeb4e246b892cd377f29eb0919" candidate_uuid: "dfe813ac860f4be68c2f87dc4bcab956" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca5cccf002874a15b99a3ec1fdea4c3f"
I20260812 09:45:19.497860 17169 raft_consensus.cc:3037] T 343ecbaeb4e246b892cd377f29eb0919 P ca5cccf002874a15b99a3ec1fdea4c3f [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:45:19.497807 16927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "343ecbaeb4e246b892cd377f29eb0919" candidate_uuid: "dfe813ac860f4be68c2f87dc4bcab956" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a386c33a01a49e4adf0dbcd90cf9223"
I20260812 09:45:19.498018 16927 raft_consensus.cc:3037] T 343ecbaeb4e246b892cd377f29eb0919 P 9a386c33a01a49e4adf0dbcd90cf9223 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:45:19.498883 17169 raft_consensus.cc:2454] T 343ecbaeb4e246b892cd377f29eb0919 P ca5cccf002874a15b99a3ec1fdea4c3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfe813ac860f4be68c2f87dc4bcab956 in term 1.
I20260812 09:45:19.499004 16927 raft_consensus.cc:2454] T 343ecbaeb4e246b892cd377f29eb0919 P 9a386c33a01a49e4adf0dbcd90cf9223 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfe813ac860f4be68c2f87dc4bcab956 in term 1.
I20260812 09:45:19.499191 17307 leader_election.cc:302] T 343ecbaeb4e246b892cd377f29eb0919 P dfe813ac860f4be68c2f87dc4bcab956 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ca5cccf002874a15b99a3ec1fdea4c3f, dfe813ac860f4be68c2f87dc4bcab956; no voters:
I20260812 09:45:19.499398 17616 raft_consensus.cc:2781] T 343ecbaeb4e246b892cd377f29eb0919 P dfe813ac860f4be68c2f87dc4bcab956 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 09:45:19.499514 17616 raft_consensus.cc:686] T 343ecbaeb4e246b892cd377f29eb0919 P dfe813ac860f4be68c2f87dc4bcab956 [term 1 LEADER]: Becoming Leader. State: Replica: dfe813ac860f4be68c2f87dc4bcab956, State: Running, Role: LEADER
I20260812 09:45:19.499656 17616 consensus_queue.cc:237] T 343ecbaeb4e246b892cd377f29eb0919 P dfe813ac860f4be68c2f87dc4bcab956 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a386c33a01a49e4adf0dbcd90cf9223" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 42111 } } peers { permanent_uuid: "ca5cccf002874a15b99a3ec1fdea4c3f" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35267 } } peers { permanent_uuid: "dfe813ac860f4be68c2f87dc4bcab956" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 35941 } }
I20260812 09:45:19.501255 16641 catalog_manager.cc:5495] T 343ecbaeb4e246b892cd377f29eb0919 P dfe813ac860f4be68c2f87dc4bcab956 reported cstate change: term changed from 0 to 1, leader changed from <none> to dfe813ac860f4be68c2f87dc4bcab956 (127.13.99.3). New cstate: current_term: 1 leader_uuid: "dfe813ac860f4be68c2f87dc4bcab956" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a386c33a01a49e4adf0dbcd90cf9223" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 42111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca5cccf002874a15b99a3ec1fdea4c3f" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfe813ac860f4be68c2f87dc4bcab956" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 35941 } health_report { overall_health: HEALTHY } } }
I20260812 09:45:19.517917 17169 raft_consensus.cc:1261] T 343ecbaeb4e246b892cd377f29eb0919 P ca5cccf002874a15b99a3ec1fdea4c3f [term 1 FOLLOWER]: Refusing update from remote peer dfe813ac860f4be68c2f87dc4bcab956: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 09:45:19.517944 16927 raft_consensus.cc:1261] T 343ecbaeb4e246b892cd377f29eb0919 P 9a386c33a01a49e4adf0dbcd90cf9223 [term 1 FOLLOWER]: Refusing update from remote peer dfe813ac860f4be68c2f87dc4bcab956: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 09:45:19.518308 17616 consensus_queue.cc:1035] T 343ecbaeb4e246b892cd377f29eb0919 P dfe813ac860f4be68c2f87dc4bcab956 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5cccf002874a15b99a3ec1fdea4c3f" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 09:45:19.518442 17619 consensus_queue.cc:1035] T 343ecbaeb4e246b892cd377f29eb0919 P dfe813ac860f4be68c2f87dc4bcab956 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a386c33a01a49e4adf0dbcd90cf9223" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 42111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 09:45:19.532085 17617 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260812 09:45:21.083369 16641 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43120:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 09:45:21.083523 16641 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43120:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 09:45:21.085978 16641 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 77dec44a5cf34f3a9c0e1dfc5ff57cd4: Sending DeleteTablet for 3 replicas of tablet 343ecbaeb4e246b892cd377f29eb0919
I20260812 09:45:21.086517 16887 tablet_service.cc:1514] Processing DeleteTablet for tablet 343ecbaeb4e246b892cd377f29eb0919 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 09:45:21 UTC) from {username='kudu', principal='kudu/127.13.99.62@KRBTEST.COM'} at 127.0.0.1:44768
I20260812 09:45:21.086596 17132 tablet_service.cc:1514] Processing DeleteTablet for tablet 343ecbaeb4e246b892cd377f29eb0919 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 09:45:21 UTC) from {username='kudu', principal='kudu/127.13.99.62@KRBTEST.COM'} at 127.0.0.1:53428
I20260812 09:45:21.086738 17382 tablet_service.cc:1514] Processing DeleteTablet for tablet 343ecbaeb4e246b892cd377f29eb0919 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 09:45:21 UTC) from {username='kudu', principal='kudu/127.13.99.62@KRBTEST.COM'} at 127.0.0.1:39042
I20260812 09:45:21.086957 17722 tablet_replica.cc:330] T 343ecbaeb4e246b892cd377f29eb0919 P ca5cccf002874a15b99a3ec1fdea4c3f: stopping tablet replica
I20260812 09:45:21.087085 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 16780
I20260812 09:45:21.087085 17722 raft_consensus.cc:2229] T 343ecbaeb4e246b892cd377f29eb0919 P ca5cccf002874a15b99a3ec1fdea4c3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 09:45:21.087096 17723 tablet_replica.cc:330] T 343ecbaeb4e246b892cd377f29eb0919 P dfe813ac860f4be68c2f87dc4bcab956: stopping tablet replica
I20260812 09:45:21.098487 17723 raft_consensus.cc:2229] T 343ecbaeb4e246b892cd377f29eb0919 P dfe813ac860f4be68c2f87dc4bcab956 [term 1 LEADER]: Raft consensus shutting down.
I20260812 09:45:21.098774 17723 raft_consensus.cc:2258] T 343ecbaeb4e246b892cd377f29eb0919 P dfe813ac860f4be68c2f87dc4bcab956 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 09:45:21.098912 17722 raft_consensus.cc:2258] T 343ecbaeb4e246b892cd377f29eb0919 P ca5cccf002874a15b99a3ec1fdea4c3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 09:45:21.100773 17723 ts_tablet_manager.cc:1905] T 343ecbaeb4e246b892cd377f29eb0919 P dfe813ac860f4be68c2f87dc4bcab956: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 09:45:21.101818 17722 ts_tablet_manager.cc:1905] T 343ecbaeb4e246b892cd377f29eb0919 P ca5cccf002874a15b99a3ec1fdea4c3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 09:45:21.103932 17722 ts_tablet_manager.cc:1918] T 343ecbaeb4e246b892cd377f29eb0919 P ca5cccf002874a15b99a3ec1fdea4c3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 09:45:21.104040 17722 log.cc:1196] T 343ecbaeb4e246b892cd377f29eb0919 P ca5cccf002874a15b99a3ec1fdea4c3f: Deleting WAL directory at /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/minicluster-data/ts-1/wal/wals/343ecbaeb4e246b892cd377f29eb0919
I20260812 09:45:21.104350 17722 ts_tablet_manager.cc:1939] T 343ecbaeb4e246b892cd377f29eb0919 P ca5cccf002874a15b99a3ec1fdea4c3f: Deleting consensus metadata
I20260812 09:45:21.104797 17723 ts_tablet_manager.cc:1918] T 343ecbaeb4e246b892cd377f29eb0919 P dfe813ac860f4be68c2f87dc4bcab956: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 09:45:21.104894 17723 log.cc:1196] T 343ecbaeb4e246b892cd377f29eb0919 P dfe813ac860f4be68c2f87dc4bcab956: Deleting WAL directory at /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786527905902985-13708-0/minicluster-data/ts-2/wal/wals/343ecbaeb4e246b892cd377f29eb0919
I20260812 09:45:21.105198 16625 catalog_manager.cc:4841] TS ca5cccf002874a15b99a3ec1fdea4c3f (127.13.99.2:35267): tablet 343ecbaeb4e246b892cd377f29eb0919 (table test-table [id=12f1a5c42ac34faaa1a5f1b4336005d8]) successfully deleted
I20260812 09:45:21.105254 17723 ts_tablet_manager.cc:1939] T 343ecbaeb4e246b892cd377f29eb0919 P dfe813ac860f4be68c2f87dc4bcab956: Deleting consensus metadata
I20260812 09:45:21.106031 16621 catalog_manager.cc:4841] TS dfe813ac860f4be68c2f87dc4bcab956 (127.13.99.3:35941): tablet 343ecbaeb4e246b892cd377f29eb0919 (table test-table [id=12f1a5c42ac34faaa1a5f1b4336005d8]) successfully deleted
W20260812 09:45:21.109965 16623 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.99.1:42111 (error 108)
I20260812 09:45:21.110176 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 17031
I20260812 09:45:21.117494 17728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:21.115999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.1:42111 (local address 127.0.0.1:44780)
0812 09:45:21.117324 (+ 1325us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:21.117396 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.1:42111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1243,"thread_start_us":785,"threads_started":1}
W20260812 09:45:21.117636 16623 catalog_manager.cc:4584] TS 9a386c33a01a49e4adf0dbcd90cf9223 (127.13.99.1:42111): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 343ecbaeb4e246b892cd377f29eb0919: Network error: Client connection negotiation failed: client connection to 127.13.99.1:42111: connect: Connection refused (error 111)
I20260812 09:45:21.123205 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 17272
I20260812 09:45:21.138178 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 16599
2026-08-12T09:45:21Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6607 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786527905902985-13708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 09:45:21 dist-test-slave-jq4t krb5kdc[17733](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 09:45:21 dist-test-slave-jq4t krb5kdc[17733](info): set up 2 sockets
Aug 12 09:45:21 dist-test-slave-jq4t krb5kdc[17733](info): commencing operation
krb5kdc: starting...
W20260812 09:45:22.428893 13708 mini_kdc.cc:121] Time spent starting KDC: real 1.251s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786527905902985-13708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786527905902985-13708-0/krb5kdc/test-user.keytab.
Aug 12 09:45:22 dist-test-slave-jq4t krb5kdc[17733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527922, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T09:45:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T09:45:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.99.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.99.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.99.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab.
Entry for principal kudu/127.13.99.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab.
I20260812 09:45:22.556447 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786527905902985-13708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786527905902985-13708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786527905902985-13708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.62:35471
--webserver_interface=127.13.99.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab
--principal=kudu/127.13.99.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:44129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.99.62:35471
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:45:22.670569 17763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:45:22.671003 17763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:45:22.671100 17763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:45:22.675459 17763 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 09:45:22.675550 17763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:45:22.675590 17763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:45:22.675626 17763 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 09:45:22.675660 17763 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 09:45:22.681105 17763 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:44129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786527905902985-13708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786527905902985-13708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.99.62:35471
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab
--principal=kudu/127.13.99.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.62:35471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.99.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786527905902985-13708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:45:22.682188 17763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:45:22.683245 17763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 09:45:22.692936 17763 server_base.cc:1034] running on GCE node
W20260812 09:45:22.692981 17772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:22.693032 17770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:22.692987 17769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:45:22.693878 17763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:45:22.694979 17763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:45:22.696188 17763 hybrid_clock.cc:648] HybridClock initialized: now 1786527922696157 us; error 44 us; skew 500 ppm
Aug 12 09:45:22 dist-test-slave-jq4t krb5kdc[17733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527922, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:45:22.699091 17763 init.cc:377] Logged in from keytab as kudu/127.13.99.62@KRBTEST.COM (short username kudu)
I20260812 09:45:22.699973 17763 webserver.cc:460] Webserver started at http://127.13.99.62:38493/ using document root <none> and password file <none>
I20260812 09:45:22.700631 17763 fs_manager.cc:359] Metadata directory not provided
I20260812 09:45:22.700730 17763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:45:22.700973 17763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:45:22.703176 17763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786527905902985-13708-0/minicluster-data/master-0/data/instance:
uuid: "eec11d81a271462bacdd71c4df85d6a2"
format_stamp: "Formatted at 2026-08-12 09:45:22 on dist-test-slave-jq4t"
server_key: "d07311c469bb61e6b361429d4b8a00f6"
server_key_iv: "b57f9c7bc99cfe3db74f1794531a162e"
server_key_version: "encryptionkey@0"
I20260812 09:45:22.703786 17763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786527905902985-13708-0/minicluster-data/master-0/wal/instance:
uuid: "eec11d81a271462bacdd71c4df85d6a2"
format_stamp: "Formatted at 2026-08-12 09:45:22 on dist-test-slave-jq4t"
server_key: "d07311c469bb61e6b361429d4b8a00f6"
server_key_iv: "b57f9c7bc99cfe3db74f1794531a162e"
server_key_version: "encryptionkey@0"
I20260812 09:45:22.707681 17763 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260812 09:45:22.710222 17779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:22.711498 17763 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260812 09:45:22.711649 17763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786527905902985-13708-0/minicluster-data/master-0/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786527905902985-13708-0/minicluster-data/master-0/wal
uuid: "eec11d81a271462bacdd71c4df85d6a2"
format_stamp: "Formatted at 2026-08-12 09:45:22 on dist-test-slave-jq4t"
server_key: "d07311c469bb61e6b361429d4b8a00f6"
server_key_iv: "b57f9c7bc99cfe3db74f1794531a162e"
server_key_version: "encryptionkey@0"
I20260812 09:45:22.711763 17763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786527905902985-13708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786527905902985-13708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786527905902985-13708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:45:22.726372 17763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:45:22.729696 17763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:45:22.729997 17763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:45:22.737344 17763 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.62:35471
I20260812 09:45:22.737454 17859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.62:35471 every 8 connection(s)
I20260812 09:45:22.738319 17763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
I20260812 09:45:22.741294 17860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:45:22.742730 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 17763
I20260812 09:45:22.742908 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786527905902985-13708-0/minicluster-data/master-0/wal/instance
I20260812 09:45:22.743216 13708 external_mini_cluster.cc:1428] Setting key fa593bee43914bcc994b68b761a02adc
I20260812 09:45:22.748260 17860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eec11d81a271462bacdd71c4df85d6a2: Bootstrap starting.
I20260812 09:45:22.750876 17860 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eec11d81a271462bacdd71c4df85d6a2: Neither blocks nor log segments found. Creating new log.
I20260812 09:45:22.751655 17860 log.cc:824] T 00000000000000000000000000000000 P eec11d81a271462bacdd71c4df85d6a2: Log is configured to *not* fsync() on all Append() calls
Aug 12 09:45:22 dist-test-slave-jq4t krb5kdc[17733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527922, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
I20260812 09:45:22.753736 17860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eec11d81a271462bacdd71c4df85d6a2: No bootstrap required, opened a new log
I20260812 09:45:22.757406 17860 raft_consensus.cc:348] T 00000000000000000000000000000000 P eec11d81a271462bacdd71c4df85d6a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eec11d81a271462bacdd71c4df85d6a2" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 35471 } }
I20260812 09:45:22.757637 17860 raft_consensus.cc:374] T 00000000000000000000000000000000 P eec11d81a271462bacdd71c4df85d6a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:45:22.757694 17860 raft_consensus.cc:729] T 00000000000000000000000000000000 P eec11d81a271462bacdd71c4df85d6a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eec11d81a271462bacdd71c4df85d6a2, State: Initialized, Role: FOLLOWER
I20260812 09:45:22.758095 17866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:22.744689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36030 (local address 127.13.99.62:35471)
0812 09:45:22.745164 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:22.745184 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:22.745227 (+ 43us) server_negotiation.cc:407] Connection header received
0812 09:45:22.746036 (+ 809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:22.746061 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:22.746483 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:22.746837 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:22.747579 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:22.749016 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:22.749887 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:22.750553 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:22.750726 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:22.753396 (+ 2670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:22.753401 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:22.753414 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:22.753440 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:22.755115 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:22.755570 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:22.755576 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:22.755580 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:22.755687 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:22.755980 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:22.755983 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:22.755985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:22.756358 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:22.756601 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:22.756918 (+ 317us) server_negotiation.cc:299] Negotiation successful
0812 09:45:22.757194 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":146,"threads_started":1}
I20260812 09:45:22.758128 17860 consensus_queue.cc:260] T 00000000000000000000000000000000 P eec11d81a271462bacdd71c4df85d6a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eec11d81a271462bacdd71c4df85d6a2" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 35471 } }
I20260812 09:45:22.758283 17860 raft_consensus.cc:388] T 00000000000000000000000000000000 P eec11d81a271462bacdd71c4df85d6a2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 09:45:22.758383 17860 raft_consensus.cc:482] T 00000000000000000000000000000000 P eec11d81a271462bacdd71c4df85d6a2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 09:45:22.758486 17860 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eec11d81a271462bacdd71c4df85d6a2 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:45:22.759472 17860 raft_consensus.cc:504] T 00000000000000000000000000000000 P eec11d81a271462bacdd71c4df85d6a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eec11d81a271462bacdd71c4df85d6a2" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 35471 } }
I20260812 09:45:22.759828 17860 leader_election.cc:302] T 00000000000000000000000000000000 P eec11d81a271462bacdd71c4df85d6a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eec11d81a271462bacdd71c4df85d6a2; no voters:
I20260812 09:45:22.760174 17860 leader_election.cc:288] T 00000000000000000000000000000000 P eec11d81a271462bacdd71c4df85d6a2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 09:45:22.760294 17871 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eec11d81a271462bacdd71c4df85d6a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 09:45:22.760560 17871 raft_consensus.cc:686] T 00000000000000000000000000000000 P eec11d81a271462bacdd71c4df85d6a2 [term 1 LEADER]: Becoming Leader. State: Replica: eec11d81a271462bacdd71c4df85d6a2, State: Running, Role: LEADER
I20260812 09:45:22.760932 17871 consensus_queue.cc:237] T 00000000000000000000000000000000 P eec11d81a271462bacdd71c4df85d6a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eec11d81a271462bacdd71c4df85d6a2" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 35471 } }
I20260812 09:45:22.761476 17860 sys_catalog.cc:564] T 00000000000000000000000000000000 P eec11d81a271462bacdd71c4df85d6a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 09:45:22.762567 17872 sys_catalog.cc:455] T 00000000000000000000000000000000 P eec11d81a271462bacdd71c4df85d6a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eec11d81a271462bacdd71c4df85d6a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eec11d81a271462bacdd71c4df85d6a2" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 35471 } } }
I20260812 09:45:22.762691 17872 sys_catalog.cc:458] T 00000000000000000000000000000000 P eec11d81a271462bacdd71c4df85d6a2 [sys.catalog]: This master's current role is: LEADER
I20260812 09:45:22.762810 17873 sys_catalog.cc:455] T 00000000000000000000000000000000 P eec11d81a271462bacdd71c4df85d6a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eec11d81a271462bacdd71c4df85d6a2. Latest consensus state: current_term: 1 leader_uuid: "eec11d81a271462bacdd71c4df85d6a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eec11d81a271462bacdd71c4df85d6a2" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 35471 } } }
I20260812 09:45:22.762990 17873 sys_catalog.cc:458] T 00000000000000000000000000000000 P eec11d81a271462bacdd71c4df85d6a2 [sys.catalog]: This master's current role is: LEADER
I20260812 09:45:22.763311 17882 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 09:45:22.766517 17882 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 09:45:22.772330 17882 catalog_manager.cc:1347] Generated new cluster ID: d99a8be639854dd994ea6605a9326838
I20260812 09:45:22.772396 17882 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 09:45:22.784524 17882 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 09:45:22.785755 17882 catalog_manager.cc:1504] Loading token signing keys...
I20260812 09:45:22.793802 17882 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eec11d81a271462bacdd71c4df85d6a2: Generated new TSK 0
I20260812 09:45:22.794741 17882 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.99.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.99.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.99.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.1.keytab.
Entry for principal kudu/127.13.99.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.1.keytab.
I20260812 09:45:22.823810 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.99.1:0
--local_ip_for_outbound_sockets=127.13.99.1
--webserver_interface=127.13.99.1
--webserver_port=0
--tserver_master_addrs=127.13.99.62:35471
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.1.keytab
--principal=kudu/127.13.99.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:44129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:45:22.938596 17907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:45:22.938846 17907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:45:22.938889 17907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:45:22.942864 17907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:45:22.942915 17907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:45:22.943007 17907 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.99.1
I20260812 09:45:22.948218 17907 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:44129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.1.keytab
--principal=kudu/127.13.99.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.99.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.99.1
--webserver_port=0
--tserver_master_addrs=127.13.99.62:35471
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17907
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.99.1
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:45:22.949237 17907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:45:22.950248 17907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:45:22.959735 17920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:22.959741 17919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:45:22.959868 17907 server_base.cc:1034] running on GCE node
W20260812 09:45:22.959735 17923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:45:22.960402 17907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:45:22.961042 17907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:45:22.962203 17907 hybrid_clock.cc:648] HybridClock initialized: now 1786527922962189 us; error 24 us; skew 500 ppm
Aug 12 09:45:22 dist-test-slave-jq4t krb5kdc[17733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527922, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:45:22.965142 17907 init.cc:377] Logged in from keytab as kudu/127.13.99.1@KRBTEST.COM (short username kudu)
I20260812 09:45:22.965988 17907 webserver.cc:460] Webserver started at http://127.13.99.1:41055/ using document root <none> and password file <none>
I20260812 09:45:22.966495 17907 fs_manager.cc:359] Metadata directory not provided
I20260812 09:45:22.966553 17907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:45:22.966712 17907 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:45:22.968688 17907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-0/data/instance:
uuid: "9f1d17246c1b46a7a56a159aa028011b"
format_stamp: "Formatted at 2026-08-12 09:45:22 on dist-test-slave-jq4t"
server_key: "2476fbedfff4b5e3d451655431d58084"
server_key_iv: "1f09260eb06e47b8276ae484b50c344f"
server_key_version: "encryptionkey@0"
I20260812 09:45:22.969206 17907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-0/wal/instance:
uuid: "9f1d17246c1b46a7a56a159aa028011b"
format_stamp: "Formatted at 2026-08-12 09:45:22 on dist-test-slave-jq4t"
server_key: "2476fbedfff4b5e3d451655431d58084"
server_key_iv: "1f09260eb06e47b8276ae484b50c344f"
server_key_version: "encryptionkey@0"
I20260812 09:45:22.972851 17907 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260812 09:45:22.975148 17934 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:22.976321 17907 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 09:45:22.976462 17907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-0/wal
uuid: "9f1d17246c1b46a7a56a159aa028011b"
format_stamp: "Formatted at 2026-08-12 09:45:22 on dist-test-slave-jq4t"
server_key: "2476fbedfff4b5e3d451655431d58084"
server_key_iv: "1f09260eb06e47b8276ae484b50c344f"
server_key_version: "encryptionkey@0"
I20260812 09:45:22.976639 17907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:45:23.015761 17907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:45:23.019006 17907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:45:23.019212 17907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:45:23.019833 17907 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 09:45:23.020812 17907 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:45:23.020885 17907 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:23.020963 17907 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:45:23.021008 17907 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:23.030334 17907 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.1:43691
I20260812 09:45:23.030421 18132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.1:43691 every 8 connection(s)
I20260812 09:45:23.031347 17907 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-0/data/info.pb
I20260812 09:45:23.040727 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 17907
I20260812 09:45:23.040884 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-0/wal/instance
I20260812 09:45:23.041173 13708 external_mini_cluster.cc:1428] Setting key 0e5cd1c7d5de9fc9fe7b4f7e1bffaaae
Aug 12 09:45:23 dist-test-slave-jq4t krb5kdc[17733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527922, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.1@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
I20260812 09:45:23.046663 17866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:23.033364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.1:34367 (local address 127.13.99.62:35471)
0812 09:45:23.033539 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:23.033545 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:23.034413 (+ 868us) server_negotiation.cc:407] Connection header received
0812 09:45:23.035755 (+ 1342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:23.035758 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:23.035810 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:23.035898 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:23.037255 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:23.037821 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:23.039030 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:23.039395 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:23.039505 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:23.042388 (+ 2883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:23.042393 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:23.042399 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:23.042427 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:23.044235 (+ 1808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:23.044898 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:23.044902 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:23.044904 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:23.045001 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:23.045448 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:23.045452 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:23.045453 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:23.045685 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:23.045756 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:23.046346 (+ 590us) server_negotiation.cc:299] Negotiation successful
0812 09:45:23.046491 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260812 09:45:23.047557 18137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:23.033774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:35471 (local address 127.13.99.1:34367)
0812 09:45:23.034244 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:23.034288 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:23.035510 (+ 1222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:23.036133 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:23.036142 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:23.036545 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:45:23.037075 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:23.037090 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:23.037956 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:23.037959 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:23.038889 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:23.038899 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:23.039601 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:23.039604 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:23.039763 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:23.040542 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:45:23.040568 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:45:23.042214 (+ 1646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:45:23.044380 (+ 2166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:23.044387 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:23.044405 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:23.044779 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:23.045109 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:23.045113 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:23.045115 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:23.045306 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:23.045820 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:45:23.045826 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:45:23.046088 (+ 262us) client_negotiation.cc:769] Sending connection context
0812 09:45:23.046286 (+ 198us) client_negotiation.cc:240] Negotiation successful
0812 09:45:23.046661 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":128,"threads_started":1}
I20260812 09:45:23.048946 18134 heartbeater.cc:347] Connected to a master server at 127.13.99.62:35471
I20260812 09:45:23.049247 18134 heartbeater.cc:464] Registering TS with master...
I20260812 09:45:23.049939 18134 heartbeater.cc:511] Master 127.13.99.62:35471 requested a full tablet report, sending...
I20260812 09:45:23.051656 17807 ts_manager.cc:194] Registered new tserver with Master: 9f1d17246c1b46a7a56a159aa028011b (127.13.99.1:43691)
WARNING: no policy specified for kudu/127.13.99.2@KRBTEST.COM; defaulting to no policy
I20260812 09:45:23.052954 17807 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.99.1@KRBTEST.COM'} at 127.13.99.1:34367
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.99.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.99.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.2.keytab.
Entry for principal kudu/127.13.99.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.2.keytab.
I20260812 09:45:23.071923 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.99.2:0
--local_ip_for_outbound_sockets=127.13.99.2
--webserver_interface=127.13.99.2
--webserver_port=0
--tserver_master_addrs=127.13.99.62:35471
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.2.keytab
--principal=kudu/127.13.99.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:44129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:45:23.185415 18142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:45:23.185665 18142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:45:23.185706 18142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:45:23.189626 18142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:45:23.189677 18142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:45:23.189731 18142 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.99.2
I20260812 09:45:23.194832 18142 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:44129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.2.keytab
--principal=kudu/127.13.99.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.99.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.99.2
--webserver_port=0
--tserver_master_addrs=127.13.99.62:35471
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18142
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.99.2
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:45:23.195818 18142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:45:23.196841 18142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:45:23.206444 18153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:23.206456 18149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:23.206456 18151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:45:23.206792 18142 server_base.cc:1034] running on GCE node
I20260812 09:45:23.207195 18142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:45:23.207844 18142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:45:23.209044 18142 hybrid_clock.cc:648] HybridClock initialized: now 1786527923209016 us; error 58 us; skew 500 ppm
Aug 12 09:45:23 dist-test-slave-jq4t krb5kdc[17733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527923, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:45:23.211899 18142 init.cc:377] Logged in from keytab as kudu/127.13.99.2@KRBTEST.COM (short username kudu)
I20260812 09:45:23.212879 18142 webserver.cc:460] Webserver started at http://127.13.99.2:42975/ using document root <none> and password file <none>
I20260812 09:45:23.213408 18142 fs_manager.cc:359] Metadata directory not provided
I20260812 09:45:23.213467 18142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:45:23.213644 18142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:45:23.215703 18142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-1/data/instance:
uuid: "fbee822505594ed4b783681f971ad5dc"
format_stamp: "Formatted at 2026-08-12 09:45:23 on dist-test-slave-jq4t"
server_key: "fa1b3857cb27f025540acf28d087645d"
server_key_iv: "bc6539eb56d6584064645067c8210d7c"
server_key_version: "encryptionkey@0"
I20260812 09:45:23.216424 18142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-1/wal/instance:
uuid: "fbee822505594ed4b783681f971ad5dc"
format_stamp: "Formatted at 2026-08-12 09:45:23 on dist-test-slave-jq4t"
server_key: "fa1b3857cb27f025540acf28d087645d"
server_key_iv: "bc6539eb56d6584064645067c8210d7c"
server_key_version: "encryptionkey@0"
I20260812 09:45:23.220165 18142 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260812 09:45:23.222726 18166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:23.224011 18142 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 09:45:23.224123 18142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-1/wal
uuid: "fbee822505594ed4b783681f971ad5dc"
format_stamp: "Formatted at 2026-08-12 09:45:23 on dist-test-slave-jq4t"
server_key: "fa1b3857cb27f025540acf28d087645d"
server_key_iv: "bc6539eb56d6584064645067c8210d7c"
server_key_version: "encryptionkey@0"
I20260812 09:45:23.224208 18142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:45:23.236469 18142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:45:23.239761 18142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:45:23.239907 18142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:45:23.240410 18142 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 09:45:23.241230 18142 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:45:23.241276 18142 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:23.241313 18142 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:45:23.241328 18142 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:23.250023 18142 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.2:42387
I20260812 09:45:23.250084 18359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.2:42387 every 8 connection(s)
I20260812 09:45:23.250929 18142 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-1/data/info.pb
I20260812 09:45:23.259048 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 18142
I20260812 09:45:23.259187 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-1/wal/instance
I20260812 09:45:23.259477 13708 external_mini_cluster.cc:1428] Setting key d031127de10dda0f7e20e502faad4e77
Aug 12 09:45:23 dist-test-slave-jq4t krb5kdc[17733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527923, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.2@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
I20260812 09:45:23.265882 17866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:23.252886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.2:51509 (local address 127.13.99.62:35471)
0812 09:45:23.253025 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:23.253030 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:23.253895 (+ 865us) server_negotiation.cc:407] Connection header received
0812 09:45:23.255175 (+ 1280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:23.255178 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:23.255229 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:23.255311 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:23.256450 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:23.257068 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:23.258290 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:23.258541 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:23.258635 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:23.261668 (+ 3033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:23.261671 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:23.261673 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:23.261697 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:23.263632 (+ 1935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:23.264171 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:23.264175 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:23.264177 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:23.264265 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:23.264730 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:23.264733 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:23.264735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:23.264938 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:23.265051 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:23.265577 (+ 526us) server_negotiation.cc:299] Negotiation successful
0812 09:45:23.265711 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260812 09:45:23.266829 18363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:23.253291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:35471 (local address 127.13.99.2:51509)
0812 09:45:23.253741 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:23.253782 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:23.254938 (+ 1156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:23.255472 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:23.255478 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:23.255830 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:45:23.256299 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:23.256312 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:23.257232 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:23.257236 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:23.258146 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:23.258156 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:23.258689 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:23.258692 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:23.258862 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:23.259601 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:45:23.259623 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:45:23.261440 (+ 1817us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:45:23.263752 (+ 2312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:23.263761 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:23.263777 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:23.264057 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:23.264410 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:23.264413 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:23.264416 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:23.264626 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:23.265051 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:45:23.265059 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:45:23.265330 (+ 271us) client_negotiation.cc:769] Sending connection context
0812 09:45:23.265534 (+ 204us) client_negotiation.cc:240] Negotiation successful
0812 09:45:23.265885 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":123,"threads_started":1}
I20260812 09:45:23.268095 18361 heartbeater.cc:347] Connected to a master server at 127.13.99.62:35471
I20260812 09:45:23.268373 18361 heartbeater.cc:464] Registering TS with master...
I20260812 09:45:23.268910 18361 heartbeater.cc:511] Master 127.13.99.62:35471 requested a full tablet report, sending...
I20260812 09:45:23.270089 17807 ts_manager.cc:194] Registered new tserver with Master: fbee822505594ed4b783681f971ad5dc (127.13.99.2:42387)
WARNING: no policy specified for kudu/127.13.99.3@KRBTEST.COM; defaulting to no policy
I20260812 09:45:23.270943 17807 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.99.2@KRBTEST.COM'} at 127.13.99.2:51509
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.99.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.99.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.3.keytab.
Entry for principal kudu/127.13.99.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.3.keytab.
I20260812 09:45:23.291262 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.99.3:0
--local_ip_for_outbound_sockets=127.13.99.3
--webserver_interface=127.13.99.3
--webserver_port=0
--tserver_master_addrs=127.13.99.62:35471
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.3.keytab
--principal=kudu/127.13.99.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:44129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:45:23.420099 18369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:45:23.420408 18369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:45:23.420485 18369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:45:23.424685 18369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:45:23.424768 18369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:45:23.424862 18369 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.99.3
I20260812 09:45:23.430513 18369 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:44129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.3.keytab
--principal=kudu/127.13.99.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.99.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.99.3
--webserver_port=0
--tserver_master_addrs=127.13.99.62:35471
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18369
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.99.3
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:45:23.431699 18369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:45:23.432865 18369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:45:23.443030 18376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:45:23.443109 18369 server_base.cc:1034] running on GCE node
W20260812 09:45:23.443037 18382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:23.443037 18377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:45:23.443693 18369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:45:23.444339 18369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:45:23.445539 18369 hybrid_clock.cc:648] HybridClock initialized: now 1786527923445524 us; error 24 us; skew 500 ppm
Aug 12 09:45:23 dist-test-slave-jq4t krb5kdc[17733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527923, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:45:23.448361 18369 init.cc:377] Logged in from keytab as kudu/127.13.99.3@KRBTEST.COM (short username kudu)
I20260812 09:45:23.449321 18369 webserver.cc:460] Webserver started at http://127.13.99.3:43735/ using document root <none> and password file <none>
I20260812 09:45:23.449909 18369 fs_manager.cc:359] Metadata directory not provided
I20260812 09:45:23.450033 18369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:45:23.450276 18369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:45:23.452360 18369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-2/data/instance:
uuid: "c4eff328c5934ed1b19580f13abc016f"
format_stamp: "Formatted at 2026-08-12 09:45:23 on dist-test-slave-jq4t"
server_key: "41fee7c8906dac14aaba46773213acc0"
server_key_iv: "3705abb7553d9445277e1a4fc4b24556"
server_key_version: "encryptionkey@0"
I20260812 09:45:23.453019 18369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-2/wal/instance:
uuid: "c4eff328c5934ed1b19580f13abc016f"
format_stamp: "Formatted at 2026-08-12 09:45:23 on dist-test-slave-jq4t"
server_key: "41fee7c8906dac14aaba46773213acc0"
server_key_iv: "3705abb7553d9445277e1a4fc4b24556"
server_key_version: "encryptionkey@0"
I20260812 09:45:23.456881 18369 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 09:45:23.459362 18390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:23.460726 18369 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 09:45:23.460865 18369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-2/wal
uuid: "c4eff328c5934ed1b19580f13abc016f"
format_stamp: "Formatted at 2026-08-12 09:45:23 on dist-test-slave-jq4t"
server_key: "41fee7c8906dac14aaba46773213acc0"
server_key_iv: "3705abb7553d9445277e1a4fc4b24556"
server_key_version: "encryptionkey@0"
I20260812 09:45:23.460979 18369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:45:23.471765 18369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:45:23.474860 18369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:45:23.475085 18369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:45:23.475683 18369 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 09:45:23.476610 18369 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:45:23.476681 18369 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:23.476753 18369 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:45:23.476807 18369 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:23.486258 18369 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.3:39335
I20260812 09:45:23.486291 18581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.3:39335 every 8 connection(s)
I20260812 09:45:23.487197 18369 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-2/data/info.pb
I20260812 09:45:23.491732 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 18369
I20260812 09:45:23.491879 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-2/wal/instance
I20260812 09:45:23.492156 13708 external_mini_cluster.cc:1428] Setting key 6bd4cde2ba47863e80906c5d183986ea
Aug 12 09:45:23 dist-test-slave-jq4t krb5kdc[17733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527923, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.3@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
I20260812 09:45:23.501338 17866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:23.489064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.3:41739 (local address 127.13.99.62:35471)
0812 09:45:23.489276 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:23.489280 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:23.490189 (+ 909us) server_negotiation.cc:407] Connection header received
0812 09:45:23.491560 (+ 1371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:23.491563 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:23.491622 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:23.491706 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:23.493107 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:23.493667 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:23.494976 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:23.495207 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:23.495333 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:23.497926 (+ 2593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:23.497930 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:23.497932 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:23.497966 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:23.499215 (+ 1249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:23.499746 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:23.499750 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:23.499752 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:23.499840 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:23.500192 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:23.500195 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:23.500197 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:23.500397 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:23.500532 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:23.501038 (+ 506us) server_negotiation.cc:299] Negotiation successful
0812 09:45:23.501164 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260812 09:45:23.502193 18584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:23.489531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:35471 (local address 127.13.99.3:41739)
0812 09:45:23.490030 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:23.490074 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:23.491344 (+ 1270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:23.491925 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:23.491933 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:23.492383 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:45:23.492954 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:23.492968 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:23.493813 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:23.493817 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:23.494836 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:23.494847 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:23.495365 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:23.495369 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:23.495543 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:23.496182 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:45:23.496199 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:45:23.497749 (+ 1550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:45:23.499357 (+ 1608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:23.499363 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:23.499378 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:23.499634 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:23.499938 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:23.499942 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:23.499944 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:23.500086 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:23.500535 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:45:23.500545 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:45:23.500786 (+ 241us) client_negotiation.cc:769] Sending connection context
0812 09:45:23.500991 (+ 205us) client_negotiation.cc:240] Negotiation successful
0812 09:45:23.501326 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":123,"threads_started":1}
I20260812 09:45:23.503369 18582 heartbeater.cc:347] Connected to a master server at 127.13.99.62:35471
I20260812 09:45:23.503623 18582 heartbeater.cc:464] Registering TS with master...
I20260812 09:45:23.504114 18582 heartbeater.cc:511] Master 127.13.99.62:35471 requested a full tablet report, sending...
I20260812 09:45:23.505259 17803 ts_manager.cc:194] Registered new tserver with Master: c4eff328c5934ed1b19580f13abc016f (127.13.99.3:39335)
I20260812 09:45:23.506001 17803 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.99.3@KRBTEST.COM'} at 127.13.99.3:41739
I20260812 09:45:23.506148 13708 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260812 09:45:23.521582 17866 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 09:45:23.520015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36038 (local address 127.13.99.62:35471)
0812 09:45:23.520154 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:23.520158 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:23.520381 (+ 223us) server_negotiation.cc:407] Connection header received
0812 09:45:23.520562 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:23.520565 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:23.520611 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:23.520692 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:23.521409 (+ 717us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36038: BlockingRecv error: recv got EOF from 127.0.0.1:36038 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 09:45:23.522925 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 17907
I20260812 09:45:23.541553 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 18142
I20260812 09:45:23.559000 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 18369
I20260812 09:45:23.571960 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 17763
2026-08-12T09:45:23Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2432 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786527905902985-13708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 09:45:23 dist-test-slave-jq4t krb5kdc[18599](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 09:45:23 dist-test-slave-jq4t krb5kdc[18599](info): set up 2 sockets
Aug 12 09:45:23 dist-test-slave-jq4t krb5kdc[18599](info): commencing operation
krb5kdc: starting...
W20260812 09:45:25.522907 13708 mini_kdc.cc:121] Time spent starting KDC: real 1.913s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786527905902985-13708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786527905902985-13708-0/krb5kdc/test-user.keytab.
Aug 12 09:45:25 dist-test-slave-jq4t krb5kdc[18599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527925, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T09:45:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T09:45:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.99.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.99.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.99.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab.
Entry for principal kudu/127.13.99.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab.
I20260812 09:45:25.720558 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786527905902985-13708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786527905902985-13708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786527905902985-13708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.62:34043
--webserver_interface=127.13.99.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab
--principal=kudu/127.13.99.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:38665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.99.62:34043
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:45:25.869908 18614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:45:25.870163 18614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:45:25.870203 18614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:45:25.874109 18614 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 09:45:25.874155 18614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:45:25.874174 18614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:45:25.874190 18614 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 09:45:25.874205 18614 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 09:45:25.879179 18614 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:38665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786527905902985-13708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786527905902985-13708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.99.62:34043
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab
--principal=kudu/127.13.99.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.62:34043
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.99.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18614
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786527905902985-13708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:45:25.880236 18614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:45:25.881239 18614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:45:25.889597 18627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:25.889599 18628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:45:25.889773 18614 server_base.cc:1034] running on GCE node
W20260812 09:45:25.889945 18630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:45:25.890559 18614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:45:25.891682 18614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:45:25.892885 18614 hybrid_clock.cc:648] HybridClock initialized: now 1786527925892864 us; error 31 us; skew 500 ppm
Aug 12 09:45:25 dist-test-slave-jq4t krb5kdc[18599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527925, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:45:25.895725 18614 init.cc:377] Logged in from keytab as kudu/127.13.99.62@KRBTEST.COM (short username kudu)
I20260812 09:45:25.896682 18614 webserver.cc:460] Webserver started at http://127.13.99.62:37895/ using document root <none> and password file <none>
I20260812 09:45:25.897254 18614 fs_manager.cc:359] Metadata directory not provided
I20260812 09:45:25.897347 18614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:45:25.897596 18614 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:45:25.899670 18614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786527905902985-13708-0/minicluster-data/master-0/data/instance:
uuid: "76edd4c00e2847cbaf6fbb6555947374"
format_stamp: "Formatted at 2026-08-12 09:45:25 on dist-test-slave-jq4t"
server_key: "03c1693b6e739f3657e06c37b9cd567f"
server_key_iv: "6dd838b032f545c850128026f4bbf588"
server_key_version: "encryptionkey@0"
I20260812 09:45:25.900243 18614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786527905902985-13708-0/minicluster-data/master-0/wal/instance:
uuid: "76edd4c00e2847cbaf6fbb6555947374"
format_stamp: "Formatted at 2026-08-12 09:45:25 on dist-test-slave-jq4t"
server_key: "03c1693b6e739f3657e06c37b9cd567f"
server_key_iv: "6dd838b032f545c850128026f4bbf588"
server_key_version: "encryptionkey@0"
I20260812 09:45:25.903904 18614 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260812 09:45:25.906539 18642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:25.907775 18614 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 09:45:25.907936 18614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786527905902985-13708-0/minicluster-data/master-0/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786527905902985-13708-0/minicluster-data/master-0/wal
uuid: "76edd4c00e2847cbaf6fbb6555947374"
format_stamp: "Formatted at 2026-08-12 09:45:25 on dist-test-slave-jq4t"
server_key: "03c1693b6e739f3657e06c37b9cd567f"
server_key_iv: "6dd838b032f545c850128026f4bbf588"
server_key_version: "encryptionkey@0"
I20260812 09:45:25.908051 18614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786527905902985-13708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786527905902985-13708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786527905902985-13708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:45:25.924897 18614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:45:25.928196 18614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:45:25.928414 18614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:45:25.935736 18614 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.62:34043
I20260812 09:45:25.935767 18732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.62:34043 every 8 connection(s)
I20260812 09:45:25.936739 18614 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
I20260812 09:45:25.939719 18734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:45:25.940874 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 18614
I20260812 09:45:25.941008 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786527905902985-13708-0/minicluster-data/master-0/wal/instance
I20260812 09:45:25.941289 13708 external_mini_cluster.cc:1428] Setting key 29eb43114459b51c7dca461d93e77c55
I20260812 09:45:25.946130 18734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76edd4c00e2847cbaf6fbb6555947374: Bootstrap starting.
I20260812 09:45:25.948426 18734 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 76edd4c00e2847cbaf6fbb6555947374: Neither blocks nor log segments found. Creating new log.
I20260812 09:45:25.949208 18734 log.cc:824] T 00000000000000000000000000000000 P 76edd4c00e2847cbaf6fbb6555947374: Log is configured to *not* fsync() on all Append() calls
Aug 12 09:45:25 dist-test-slave-jq4t krb5kdc[18599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527925, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
I20260812 09:45:25.951548 18734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76edd4c00e2847cbaf6fbb6555947374: No bootstrap required, opened a new log
I20260812 09:45:25.955895 18734 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76edd4c00e2847cbaf6fbb6555947374 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76edd4c00e2847cbaf6fbb6555947374" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 34043 } }
I20260812 09:45:25.956144 18734 raft_consensus.cc:374] T 00000000000000000000000000000000 P 76edd4c00e2847cbaf6fbb6555947374 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:45:25.956238 18738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:25.942754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41020 (local address 127.13.99.62:34043)
0812 09:45:25.943242 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:25.943257 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:25.943295 (+ 38us) server_negotiation.cc:407] Connection header received
0812 09:45:25.943947 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:25.943963 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:25.944322 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:25.944701 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:25.945568 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:25.946639 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:25.947516 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:25.948005 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:25.948186 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:25.951128 (+ 2942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:25.951133 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:25.951148 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:25.951174 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:25.953206 (+ 2032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:25.953632 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:25.953638 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:25.953644 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:25.953797 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:25.954152 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:25.954158 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:25.954163 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:25.954581 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:25.954826 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:25.955106 (+ 280us) server_negotiation.cc:299] Negotiation successful
0812 09:45:25.955448 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":143,"threads_started":1}
I20260812 09:45:25.956228 18734 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76edd4c00e2847cbaf6fbb6555947374 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76edd4c00e2847cbaf6fbb6555947374, State: Initialized, Role: FOLLOWER
I20260812 09:45:25.956840 18734 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76edd4c00e2847cbaf6fbb6555947374 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76edd4c00e2847cbaf6fbb6555947374" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 34043 } }
I20260812 09:45:25.957005 18734 raft_consensus.cc:388] T 00000000000000000000000000000000 P 76edd4c00e2847cbaf6fbb6555947374 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 09:45:25.957103 18734 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76edd4c00e2847cbaf6fbb6555947374 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 09:45:25.957204 18734 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76edd4c00e2847cbaf6fbb6555947374 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:45:25.958153 18734 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76edd4c00e2847cbaf6fbb6555947374 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76edd4c00e2847cbaf6fbb6555947374" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 34043 } }
I20260812 09:45:25.958511 18734 leader_election.cc:302] T 00000000000000000000000000000000 P 76edd4c00e2847cbaf6fbb6555947374 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76edd4c00e2847cbaf6fbb6555947374; no voters:
I20260812 09:45:25.958873 18734 leader_election.cc:288] T 00000000000000000000000000000000 P 76edd4c00e2847cbaf6fbb6555947374 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 09:45:25.958958 18741 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76edd4c00e2847cbaf6fbb6555947374 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 09:45:25.959185 18741 raft_consensus.cc:686] T 00000000000000000000000000000000 P 76edd4c00e2847cbaf6fbb6555947374 [term 1 LEADER]: Becoming Leader. State: Replica: 76edd4c00e2847cbaf6fbb6555947374, State: Running, Role: LEADER
I20260812 09:45:25.959277 13708 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 09:45:25.959520 18741 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76edd4c00e2847cbaf6fbb6555947374 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76edd4c00e2847cbaf6fbb6555947374" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 34043 } }
I20260812 09:45:25.960145 18734 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76edd4c00e2847cbaf6fbb6555947374 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 09:45:25.961243 18743 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76edd4c00e2847cbaf6fbb6555947374 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "76edd4c00e2847cbaf6fbb6555947374" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76edd4c00e2847cbaf6fbb6555947374" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 34043 } } }
I20260812 09:45:25.961403 18743 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76edd4c00e2847cbaf6fbb6555947374 [sys.catalog]: This master's current role is: LEADER
I20260812 09:45:25.961664 18743 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76edd4c00e2847cbaf6fbb6555947374 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76edd4c00e2847cbaf6fbb6555947374. Latest consensus state: current_term: 1 leader_uuid: "76edd4c00e2847cbaf6fbb6555947374" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76edd4c00e2847cbaf6fbb6555947374" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 34043 } } }
I20260812 09:45:25.961812 18743 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76edd4c00e2847cbaf6fbb6555947374 [sys.catalog]: This master's current role is: LEADER
I20260812 09:45:25.962150 18754 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 09:45:25.965842 18754 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 09:45:25.971747 18754 catalog_manager.cc:1347] Generated new cluster ID: d95911c6303a4f80b93c915152312b62
I20260812 09:45:25.971817 18754 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 09:45:25.981510 18738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:25.975648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41030 (local address 127.13.99.62:34043)
0812 09:45:25.975755 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:25.975759 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:25.976043 (+ 284us) server_negotiation.cc:407] Connection header received
0812 09:45:25.976171 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:25.976175 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:25.976237 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:25.976333 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:25.976939 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:25.977778 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:25.978947 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:25.979333 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:25.979449 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:25.980404 (+ 955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:25.980408 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:25.980411 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 09:45:25.981028 (+ 617us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:25.981063 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 09:45:25.981083 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:25.981177 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:25.981285 (+ 108us) server_negotiation.cc:299] Negotiation successful
0812 09:45:25.981348 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260812 09:45:25.989425 18754 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 09:45:25.990784 18754 catalog_manager.cc:1504] Loading token signing keys...
I20260812 09:45:25.998597 18754 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 76edd4c00e2847cbaf6fbb6555947374: Generated new TSK 0
I20260812 09:45:25.999446 18754 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 09:45:26.008392 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 18614
2026-08-12T09:45:26Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2426 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786527905902985-13708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 09:45:26 dist-test-slave-jq4t krb5kdc[18792](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 09:45:26 dist-test-slave-jq4t krb5kdc[18792](info): set up 2 sockets
Aug 12 09:45:26 dist-test-slave-jq4t krb5kdc[18792](info): commencing operation
krb5kdc: starting...
W20260812 09:45:27.290025 13708 mini_kdc.cc:121] Time spent starting KDC: real 1.253s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786527905902985-13708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786527905902985-13708-0/krb5kdc/test-user.keytab.
Aug 12 09:45:27 dist-test-slave-jq4t krb5kdc[18792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527927, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T09:45:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T09:45:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.99.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.99.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.99.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab.
Entry for principal kudu/127.13.99.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab.
I20260812 09:45:27.419489 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786527905902985-13708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786527905902985-13708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786527905902985-13708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.62:39691
--webserver_interface=127.13.99.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab
--principal=kudu/127.13.99.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:44793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.99.62:39691
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:45:27.533334 18817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:45:27.533613 18817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:45:27.533656 18817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:45:27.537645 18817 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 09:45:27.537719 18817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:45:27.537738 18817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:45:27.537756 18817 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 09:45:27.537771 18817 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 09:45:27.543141 18817 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:44793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786527905902985-13708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786527905902985-13708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.99.62:39691
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab
--principal=kudu/127.13.99.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.62:39691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.99.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18817
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786527905902985-13708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:45:27.544163 18817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:45:27.545204 18817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:45:27.554090 18828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:27.554183 18825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:45:27.554147 18817 server_base.cc:1034] running on GCE node
W20260812 09:45:27.554096 18830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:45:27.554939 18817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:45:27.556043 18817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:45:27.557260 18817 hybrid_clock.cc:648] HybridClock initialized: now 1786527927557239 us; error 31 us; skew 500 ppm
Aug 12 09:45:27 dist-test-slave-jq4t krb5kdc[18792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527927, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:45:27.563421 18817 init.cc:377] Logged in from keytab as kudu/127.13.99.62@KRBTEST.COM (short username kudu)
I20260812 09:45:27.564445 18817 webserver.cc:460] Webserver started at http://127.13.99.62:41863/ using document root <none> and password file <none>
I20260812 09:45:27.565124 18817 fs_manager.cc:359] Metadata directory not provided
I20260812 09:45:27.565223 18817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:45:27.565476 18817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:45:27.567647 18817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786527905902985-13708-0/minicluster-data/master-0/data/instance:
uuid: "4eb509bcc6c94330b08ae5d59a255444"
format_stamp: "Formatted at 2026-08-12 09:45:27 on dist-test-slave-jq4t"
server_key: "265816f2ab719fa1847981fd650332ad"
server_key_iv: "fe1a6fc7698e8da54e8e0bb9fdeec79d"
server_key_version: "encryptionkey@0"
I20260812 09:45:27.568274 18817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786527905902985-13708-0/minicluster-data/master-0/wal/instance:
uuid: "4eb509bcc6c94330b08ae5d59a255444"
format_stamp: "Formatted at 2026-08-12 09:45:27 on dist-test-slave-jq4t"
server_key: "265816f2ab719fa1847981fd650332ad"
server_key_iv: "fe1a6fc7698e8da54e8e0bb9fdeec79d"
server_key_version: "encryptionkey@0"
I20260812 09:45:27.572160 18817 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260812 09:45:27.574801 18838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:27.576030 18817 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 09:45:27.576184 18817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786527905902985-13708-0/minicluster-data/master-0/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786527905902985-13708-0/minicluster-data/master-0/wal
uuid: "4eb509bcc6c94330b08ae5d59a255444"
format_stamp: "Formatted at 2026-08-12 09:45:27 on dist-test-slave-jq4t"
server_key: "265816f2ab719fa1847981fd650332ad"
server_key_iv: "fe1a6fc7698e8da54e8e0bb9fdeec79d"
server_key_version: "encryptionkey@0"
I20260812 09:45:27.576306 18817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786527905902985-13708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786527905902985-13708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786527905902985-13708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:45:27.589182 18817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:45:27.592319 18817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:45:27.592613 18817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:45:27.600178 18817 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.62:39691
I20260812 09:45:27.600596 18926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.62:39691 every 8 connection(s)
I20260812 09:45:27.601260 18817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
I20260812 09:45:27.604230 18927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:45:27.605895 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 18817
I20260812 09:45:27.606045 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786527905902985-13708-0/minicluster-data/master-0/wal/instance
I20260812 09:45:27.606360 13708 external_mini_cluster.cc:1428] Setting key 0c723cd8815bb58bae53abd74f291887
I20260812 09:45:27.611131 18927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4eb509bcc6c94330b08ae5d59a255444: Bootstrap starting.
I20260812 09:45:27.613898 18927 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4eb509bcc6c94330b08ae5d59a255444: Neither blocks nor log segments found. Creating new log.
I20260812 09:45:27.614866 18927 log.cc:824] T 00000000000000000000000000000000 P 4eb509bcc6c94330b08ae5d59a255444: Log is configured to *not* fsync() on all Append() calls
Aug 12 09:45:27 dist-test-slave-jq4t krb5kdc[18792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527927, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
I20260812 09:45:27.617249 18927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4eb509bcc6c94330b08ae5d59a255444: No bootstrap required, opened a new log
I20260812 09:45:27.622180 18931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:27.607832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40592 (local address 127.13.99.62:39691)
0812 09:45:27.608275 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:27.608292 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:27.608330 (+ 38us) server_negotiation.cc:407] Connection header received
0812 09:45:27.609033 (+ 703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:27.609051 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:27.609628 (+ 577us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:27.609961 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:27.610878 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:27.611997 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:27.613325 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:27.613817 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:27.613952 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:27.616820 (+ 2868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:27.616827 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:27.616844 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:27.616882 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:27.618865 (+ 1983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:27.619314 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:27.619321 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:27.619326 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:27.619488 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:27.619772 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:27.619775 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:27.619777 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:27.620342 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:27.620569 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:27.620868 (+ 299us) server_negotiation.cc:299] Negotiation successful
0812 09:45:27.621182 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":147,"threads_started":1}
I20260812 09:45:27.622228 18927 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4eb509bcc6c94330b08ae5d59a255444 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4eb509bcc6c94330b08ae5d59a255444" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 39691 } }
I20260812 09:45:27.622504 18927 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4eb509bcc6c94330b08ae5d59a255444 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:45:27.622578 18927 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4eb509bcc6c94330b08ae5d59a255444 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb509bcc6c94330b08ae5d59a255444, State: Initialized, Role: FOLLOWER
I20260812 09:45:27.623075 18927 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4eb509bcc6c94330b08ae5d59a255444 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4eb509bcc6c94330b08ae5d59a255444" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 39691 } }
I20260812 09:45:27.623235 18927 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4eb509bcc6c94330b08ae5d59a255444 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 09:45:27.623313 18927 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4eb509bcc6c94330b08ae5d59a255444 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 09:45:27.623415 18927 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4eb509bcc6c94330b08ae5d59a255444 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:45:27.624392 18927 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4eb509bcc6c94330b08ae5d59a255444 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4eb509bcc6c94330b08ae5d59a255444" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 39691 } }
I20260812 09:45:27.624766 18927 leader_election.cc:302] T 00000000000000000000000000000000 P 4eb509bcc6c94330b08ae5d59a255444 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4eb509bcc6c94330b08ae5d59a255444; no voters:
I20260812 09:45:27.625099 18927 leader_election.cc:288] T 00000000000000000000000000000000 P 4eb509bcc6c94330b08ae5d59a255444 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 09:45:27.625182 18933 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4eb509bcc6c94330b08ae5d59a255444 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 09:45:27.625432 18933 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4eb509bcc6c94330b08ae5d59a255444 [term 1 LEADER]: Becoming Leader. State: Replica: 4eb509bcc6c94330b08ae5d59a255444, State: Running, Role: LEADER
I20260812 09:45:27.625730 18933 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4eb509bcc6c94330b08ae5d59a255444 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4eb509bcc6c94330b08ae5d59a255444" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 39691 } }
I20260812 09:45:27.626368 18927 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4eb509bcc6c94330b08ae5d59a255444 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 09:45:27.627417 18934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4eb509bcc6c94330b08ae5d59a255444 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4eb509bcc6c94330b08ae5d59a255444" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4eb509bcc6c94330b08ae5d59a255444" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 39691 } } }
I20260812 09:45:27.627573 18934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4eb509bcc6c94330b08ae5d59a255444 [sys.catalog]: This master's current role is: LEADER
I20260812 09:45:27.627929 18949 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 09:45:27.627899 18937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4eb509bcc6c94330b08ae5d59a255444 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4eb509bcc6c94330b08ae5d59a255444. Latest consensus state: current_term: 1 leader_uuid: "4eb509bcc6c94330b08ae5d59a255444" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4eb509bcc6c94330b08ae5d59a255444" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 39691 } } }
I20260812 09:45:27.628588 18937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4eb509bcc6c94330b08ae5d59a255444 [sys.catalog]: This master's current role is: LEADER
I20260812 09:45:27.631359 18949 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 09:45:27.637192 18949 catalog_manager.cc:1347] Generated new cluster ID: b10d063a165f4c4ca6fe3658f049a011
I20260812 09:45:27.637270 18949 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 09:45:27.648036 18949 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 09:45:27.649333 18949 catalog_manager.cc:1504] Loading token signing keys...
I20260812 09:45:27.655685 18949 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4eb509bcc6c94330b08ae5d59a255444: Generated new TSK 0
I20260812 09:45:27.656447 18949 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.99.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.99.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.99.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.1.keytab.
Entry for principal kudu/127.13.99.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.1.keytab.
I20260812 09:45:27.692313 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.99.1:0
--local_ip_for_outbound_sockets=127.13.99.1
--webserver_interface=127.13.99.1
--webserver_port=0
--tserver_master_addrs=127.13.99.62:39691
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.1.keytab
--principal=kudu/127.13.99.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:44793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:45:27.814479 18965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:45:27.814741 18965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:45:27.814783 18965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:45:27.818898 18965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:45:27.818964 18965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:45:27.819046 18965 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.99.1
I20260812 09:45:27.824301 18965 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:44793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.1.keytab
--principal=kudu/127.13.99.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.99.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.99.1
--webserver_port=0
--tserver_master_addrs=127.13.99.62:39691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18965
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.99.1
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:45:27.825518 18965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:45:27.826588 18965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:45:27.836601 18981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:45:27.836689 18965 server_base.cc:1034] running on GCE node
W20260812 09:45:27.836593 18979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:27.836593 18977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:45:27.837292 18965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:45:27.837970 18965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:45:27.839160 18965 hybrid_clock.cc:648] HybridClock initialized: now 1786527927839152 us; error 48 us; skew 500 ppm
Aug 12 09:45:27 dist-test-slave-jq4t krb5kdc[18792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527927, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:45:27.842114 18965 init.cc:377] Logged in from keytab as kudu/127.13.99.1@KRBTEST.COM (short username kudu)
I20260812 09:45:27.843037 18965 webserver.cc:460] Webserver started at http://127.13.99.1:45115/ using document root <none> and password file <none>
I20260812 09:45:27.843626 18965 fs_manager.cc:359] Metadata directory not provided
I20260812 09:45:27.843722 18965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:45:27.843969 18965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:45:27.846103 18965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-0/data/instance:
uuid: "2995a8993d184327b0a25e833de16c66"
format_stamp: "Formatted at 2026-08-12 09:45:27 on dist-test-slave-jq4t"
server_key: "aee7593357098241a18d21f98c500b83"
server_key_iv: "d1568f2ce52416698da69c5df6f65ee2"
server_key_version: "encryptionkey@0"
I20260812 09:45:27.846692 18965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-0/wal/instance:
uuid: "2995a8993d184327b0a25e833de16c66"
format_stamp: "Formatted at 2026-08-12 09:45:27 on dist-test-slave-jq4t"
server_key: "aee7593357098241a18d21f98c500b83"
server_key_iv: "d1568f2ce52416698da69c5df6f65ee2"
server_key_version: "encryptionkey@0"
I20260812 09:45:27.850499 18965 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 09:45:27.853008 18991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:27.854233 18965 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 09:45:27.854384 18965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-0/wal
uuid: "2995a8993d184327b0a25e833de16c66"
format_stamp: "Formatted at 2026-08-12 09:45:27 on dist-test-slave-jq4t"
server_key: "aee7593357098241a18d21f98c500b83"
server_key_iv: "d1568f2ce52416698da69c5df6f65ee2"
server_key_version: "encryptionkey@0"
I20260812 09:45:27.854498 18965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:45:27.875659 18965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:45:27.878770 18965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:45:27.878988 18965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:45:27.879681 18965 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 09:45:27.880637 18965 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:45:27.880735 18965 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:27.880810 18965 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:45:27.880841 18965 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:27.889885 18965 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.1:37215
I20260812 09:45:27.889914 19168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.1:37215 every 8 connection(s)
I20260812 09:45:27.890825 18965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-0/data/info.pb
I20260812 09:45:27.891081 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 18965
I20260812 09:45:27.891211 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-0/wal/instance
I20260812 09:45:27.891562 13708 external_mini_cluster.cc:1428] Setting key 84cd73197d23a86b8ba70bd3a67a21a9
Aug 12 09:45:27 dist-test-slave-jq4t krb5kdc[18792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527927, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.1@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.99.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.99.2@KRBTEST.COM" created.
I20260812 09:45:27.907508 18931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:27.892885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.1:48161 (local address 127.13.99.62:39691)
0812 09:45:27.893041 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:27.893051 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:27.894098 (+ 1047us) server_negotiation.cc:407] Connection header received
0812 09:45:27.896038 (+ 1940us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:27.896043 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:27.896106 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:27.896204 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:27.897699 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:27.898240 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:27.900231 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:27.900472 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:27.900634 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:27.903622 (+ 2988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:27.903629 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:27.903632 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:27.903663 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:27.905167 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:27.905759 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:27.905765 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:27.905772 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:27.905890 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:27.906256 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:27.906259 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:27.906260 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:27.906555 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:27.906670 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:27.907269 (+ 599us) server_negotiation.cc:299] Negotiation successful
0812 09:45:27.907382 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260812 09:45:27.908803 19172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:27.893372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:39691 (local address 127.13.99.1:48161)
0812 09:45:27.893910 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:27.893973 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:27.895816 (+ 1843us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:27.896378 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:27.896385 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:27.896814 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:45:27.897531 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:27.897554 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:27.898399 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:27.898406 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:27.900068 (+ 1662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:27.900086 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:27.900701 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:27.900708 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:27.900964 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:27.901650 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:45:27.901672 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:45:27.903421 (+ 1749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:45:27.905297 (+ 1876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:27.905303 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:27.905319 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:27.905622 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:27.906021 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:27.906024 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:27.906025 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:27.906166 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:27.906682 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:45:27.906691 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:45:27.906979 (+ 288us) client_negotiation.cc:769] Sending connection context
0812 09:45:27.907211 (+ 232us) client_negotiation.cc:240] Negotiation successful
0812 09:45:27.907620 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":119,"threads_started":1}
I20260812 09:45:27.910323 19169 heartbeater.cc:347] Connected to a master server at 127.13.99.62:39691
I20260812 09:45:27.910591 19169 heartbeater.cc:464] Registering TS with master...
I20260812 09:45:27.911115 19169 heartbeater.cc:511] Master 127.13.99.62:39691 requested a full tablet report, sending...
I20260812 09:45:27.912993 18865 ts_manager.cc:194] Registered new tserver with Master: 2995a8993d184327b0a25e833de16c66 (127.13.99.1:37215)
I20260812 09:45:27.914712 18865 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.99.1@KRBTEST.COM'} at 127.13.99.1:48161
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.99.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.2.keytab.
Entry for principal kudu/127.13.99.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.2.keytab.
I20260812 09:45:27.924032 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.99.2:0
--local_ip_for_outbound_sockets=127.13.99.2
--webserver_interface=127.13.99.2
--webserver_port=0
--tserver_master_addrs=127.13.99.62:39691
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.2.keytab
--principal=kudu/127.13.99.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:44793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:45:28.038283 19176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:45:28.038532 19176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:45:28.038574 19176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:45:28.042557 19176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:45:28.042618 19176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:45:28.042675 19176 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.99.2
I20260812 09:45:28.047899 19176 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:44793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.2.keytab
--principal=kudu/127.13.99.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.99.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.99.2
--webserver_port=0
--tserver_master_addrs=127.13.99.62:39691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19176
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.99.2
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:45:28.049029 19176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:45:28.050060 19176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:45:28.060925 19186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:28.060973 19183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:45:28.061108 19176 server_base.cc:1034] running on GCE node
W20260812 09:45:28.061332 19182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:45:28.061753 19176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:45:28.062470 19176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:45:28.063654 19176 hybrid_clock.cc:648] HybridClock initialized: now 1786527928063616 us; error 48 us; skew 500 ppm
Aug 12 09:45:28 dist-test-slave-jq4t krb5kdc[18792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527928, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:45:28.066560 19176 init.cc:377] Logged in from keytab as kudu/127.13.99.2@KRBTEST.COM (short username kudu)
I20260812 09:45:28.067546 19176 webserver.cc:460] Webserver started at http://127.13.99.2:34521/ using document root <none> and password file <none>
I20260812 09:45:28.068138 19176 fs_manager.cc:359] Metadata directory not provided
I20260812 09:45:28.068235 19176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:45:28.068514 19176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:45:28.070631 19176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-1/data/instance:
uuid: "862939b30ad3441d9d7108f45fd0a918"
format_stamp: "Formatted at 2026-08-12 09:45:28 on dist-test-slave-jq4t"
server_key: "cb11fff5d0e146b91d12d62ada68d474"
server_key_iv: "6c9460cb414d6394f07a2f493eb277db"
server_key_version: "encryptionkey@0"
I20260812 09:45:28.071250 19176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-1/wal/instance:
uuid: "862939b30ad3441d9d7108f45fd0a918"
format_stamp: "Formatted at 2026-08-12 09:45:28 on dist-test-slave-jq4t"
server_key: "cb11fff5d0e146b91d12d62ada68d474"
server_key_iv: "6c9460cb414d6394f07a2f493eb277db"
server_key_version: "encryptionkey@0"
I20260812 09:45:28.075066 19176 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 09:45:28.077574 19206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:28.079121 19176 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 09:45:28.079324 19176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-1/wal
uuid: "862939b30ad3441d9d7108f45fd0a918"
format_stamp: "Formatted at 2026-08-12 09:45:28 on dist-test-slave-jq4t"
server_key: "cb11fff5d0e146b91d12d62ada68d474"
server_key_iv: "6c9460cb414d6394f07a2f493eb277db"
server_key_version: "encryptionkey@0"
I20260812 09:45:28.079457 19176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:45:28.099716 19176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:45:28.102800 19176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:45:28.103027 19176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:45:28.103654 19176 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 09:45:28.104615 19176 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:45:28.104686 19176 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:28.104753 19176 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:45:28.104804 19176 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:28.114265 19176 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.2:35171
I20260812 09:45:28.114292 19393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.2:35171 every 8 connection(s)
I20260812 09:45:28.115167 19176 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-1/data/info.pb
I20260812 09:45:28.121619 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 19176
I20260812 09:45:28.121735 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-1/wal/instance
I20260812 09:45:28.122010 13708 external_mini_cluster.cc:1428] Setting key e13bd5dffacb6c933738fc00f042fe5e
Aug 12 09:45:28 dist-test-slave-jq4t krb5kdc[18792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527928, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.2@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
I20260812 09:45:28.130199 18931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:28.117063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.2:60819 (local address 127.13.99.62:39691)
0812 09:45:28.117208 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:28.117212 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:28.118047 (+ 835us) server_negotiation.cc:407] Connection header received
0812 09:45:28.119453 (+ 1406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:28.119456 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:28.119517 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:28.119624 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:28.120891 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:28.121519 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:28.122820 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:28.123114 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:28.123226 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:28.125979 (+ 2753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:28.125982 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:28.125984 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:28.126008 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:28.127803 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:28.128562 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:28.128565 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:28.128566 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:28.128684 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:28.129073 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:28.129076 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:28.129078 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:28.129270 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:28.129385 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:28.129895 (+ 510us) server_negotiation.cc:299] Negotiation successful
0812 09:45:28.130027 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260812 09:45:28.130982 19396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:28.117446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:39691 (local address 127.13.99.2:60819)
0812 09:45:28.117897 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:28.117940 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:28.119216 (+ 1276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:28.119807 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:28.119814 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:28.120180 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:45:28.120732 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:28.120749 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:28.121654 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:28.121657 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:28.122700 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:28.122711 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:28.123379 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:28.123382 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:28.123525 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:28.124142 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:45:28.124159 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:45:28.125803 (+ 1644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:45:28.127922 (+ 2119us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:28.127931 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:28.127948 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:28.128364 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:28.128818 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:28.128820 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:28.128821 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:28.128958 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:28.129390 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:45:28.129400 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:45:28.129640 (+ 240us) client_negotiation.cc:769] Sending connection context
0812 09:45:28.129828 (+ 188us) client_negotiation.cc:240] Negotiation successful
0812 09:45:28.130149 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":116,"threads_started":1}
I20260812 09:45:28.132185 19394 heartbeater.cc:347] Connected to a master server at 127.13.99.62:39691
I20260812 09:45:28.132541 19394 heartbeater.cc:464] Registering TS with master...
I20260812 09:45:28.133131 19394 heartbeater.cc:511] Master 127.13.99.62:39691 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.13.99.3@KRBTEST.COM; defaulting to no policy
I20260812 09:45:28.134332 18865 ts_manager.cc:194] Registered new tserver with Master: 862939b30ad3441d9d7108f45fd0a918 (127.13.99.2:35171)
I20260812 09:45:28.134879 18865 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.99.2@KRBTEST.COM'} at 127.13.99.2:60819
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.99.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.99.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.3.keytab.
Entry for principal kudu/127.13.99.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.3.keytab.
I20260812 09:45:28.152004 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.99.3:0
--local_ip_for_outbound_sockets=127.13.99.3
--webserver_interface=127.13.99.3
--webserver_port=0
--tserver_master_addrs=127.13.99.62:39691
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.3.keytab
--principal=kudu/127.13.99.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:44793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:45:28.268159 19400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:45:28.268383 19400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:45:28.268453 19400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:45:28.272389 19400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:45:28.272442 19400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:45:28.272570 19400 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.99.3
I20260812 09:45:28.277697 19400 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:44793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.3.keytab
--principal=kudu/127.13.99.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.99.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.99.3
--webserver_port=0
--tserver_master_addrs=127.13.99.62:39691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19400
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.99.3
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:45:28.278689 19400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:45:28.279805 19400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:45:28.289739 19410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:45:28.289776 19400 server_base.cc:1034] running on GCE node
W20260812 09:45:28.289685 19409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:28.289685 19413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:45:28.290282 19400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:45:28.290917 19400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:45:28.292109 19400 hybrid_clock.cc:648] HybridClock initialized: now 1786527928292094 us; error 28 us; skew 500 ppm
Aug 12 09:45:28 dist-test-slave-jq4t krb5kdc[18792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527928, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:45:28.294865 19400 init.cc:377] Logged in from keytab as kudu/127.13.99.3@KRBTEST.COM (short username kudu)
I20260812 09:45:28.295717 19400 webserver.cc:460] Webserver started at http://127.13.99.3:38391/ using document root <none> and password file <none>
I20260812 09:45:28.296233 19400 fs_manager.cc:359] Metadata directory not provided
I20260812 09:45:28.296289 19400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:45:28.296453 19400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:45:28.298519 19400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-2/data/instance:
uuid: "afb6047f8ede4eaf87aaad098b5180bd"
format_stamp: "Formatted at 2026-08-12 09:45:28 on dist-test-slave-jq4t"
server_key: "30e1b945b0232c91f9ce4947b269ece4"
server_key_iv: "f417009e998b248609a12ca283561ef1"
server_key_version: "encryptionkey@0"
I20260812 09:45:28.299037 19400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-2/wal/instance:
uuid: "afb6047f8ede4eaf87aaad098b5180bd"
format_stamp: "Formatted at 2026-08-12 09:45:28 on dist-test-slave-jq4t"
server_key: "30e1b945b0232c91f9ce4947b269ece4"
server_key_iv: "f417009e998b248609a12ca283561ef1"
server_key_version: "encryptionkey@0"
I20260812 09:45:28.302659 19400 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260812 09:45:28.304962 19425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:28.306159 19400 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 09:45:28.306252 19400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-2/wal
uuid: "afb6047f8ede4eaf87aaad098b5180bd"
format_stamp: "Formatted at 2026-08-12 09:45:28 on dist-test-slave-jq4t"
server_key: "30e1b945b0232c91f9ce4947b269ece4"
server_key_iv: "f417009e998b248609a12ca283561ef1"
server_key_version: "encryptionkey@0"
I20260812 09:45:28.306384 19400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:45:28.324625 19400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:45:28.327998 19400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:45:28.328258 19400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:45:28.329021 19400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 09:45:28.329945 19400 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:45:28.329998 19400 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:28.330042 19400 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:45:28.330106 19400 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:28.340524 19400 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.3:36561
I20260812 09:45:28.340557 19607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.3:36561 every 8 connection(s)
I20260812 09:45:28.341513 19400 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-2/data/info.pb
I20260812 09:45:28.348923 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 19400
I20260812 09:45:28.349084 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786527905902985-13708-0/minicluster-data/ts-2/wal/instance
I20260812 09:45:28.349360 13708 external_mini_cluster.cc:1428] Setting key 1acb936f9a0906bbd3e4636d9843c6ce
Aug 12 09:45:28 dist-test-slave-jq4t krb5kdc[18792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527928, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.3@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
I20260812 09:45:28.355695 18931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:28.343490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.3:43077 (local address 127.13.99.62:39691)
0812 09:45:28.343647 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:28.343651 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:28.344516 (+ 865us) server_negotiation.cc:407] Connection header received
0812 09:45:28.345860 (+ 1344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:28.345864 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:28.345927 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:28.346019 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:28.347250 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:28.347815 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:28.349040 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:28.349333 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:28.349457 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:28.352422 (+ 2965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:28.352426 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:28.352429 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:28.352459 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:28.353794 (+ 1335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:28.354276 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:28.354279 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:28.354280 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:28.354378 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:28.354678 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:28.354680 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:28.354681 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:28.354857 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:28.354942 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:28.355451 (+ 509us) server_negotiation.cc:299] Negotiation successful
0812 09:45:28.355573 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260812 09:45:28.356593 19610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:28.343808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:39691 (local address 127.13.99.3:43077)
0812 09:45:28.344311 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:28.344355 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:28.345627 (+ 1272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:28.346197 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:28.346204 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:28.346578 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:45:28.347082 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:28.347096 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:28.347956 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:28.347960 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:28.348900 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:28.348910 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:28.349506 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:28.349510 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:28.349724 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:28.350485 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:45:28.350513 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:45:28.352236 (+ 1723us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:45:28.353901 (+ 1665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:28.353906 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:28.353918 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:28.354182 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:28.354462 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:28.354464 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:28.354466 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:28.354597 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:28.354951 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:45:28.354955 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:45:28.355203 (+ 248us) client_negotiation.cc:769] Sending connection context
0812 09:45:28.355363 (+ 160us) client_negotiation.cc:240] Negotiation successful
0812 09:45:28.355623 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":131,"threads_started":1}
I20260812 09:45:28.357750 19608 heartbeater.cc:347] Connected to a master server at 127.13.99.62:39691
I20260812 09:45:28.358029 19608 heartbeater.cc:464] Registering TS with master...
I20260812 09:45:28.358593 19608 heartbeater.cc:511] Master 127.13.99.62:39691 requested a full tablet report, sending...
I20260812 09:45:28.359721 18865 ts_manager.cc:194] Registered new tserver with Master: afb6047f8ede4eaf87aaad098b5180bd (127.13.99.3:36561)
I20260812 09:45:28.360359 18865 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.99.3@KRBTEST.COM'} at 127.13.99.3:43077
I20260812 09:45:28.363587 13708 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 09:45:28 dist-test-slave-jq4t krb5kdc[18792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527928, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 12 09:45:28 dist-test-slave-jq4t krb5kdc[18792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527928, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
I20260812 09:45:28.390264 18931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:28.381937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40604 (local address 127.13.99.62:39691)
0812 09:45:28.382109 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:28.382114 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:28.382282 (+ 168us) server_negotiation.cc:407] Connection header received
0812 09:45:28.382380 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:28.382383 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:28.382428 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:28.382500 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:28.383260 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:28.383779 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:28.384714 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:28.384925 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:28.385024 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:28.387411 (+ 2387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:28.387414 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:28.387416 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:28.387439 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:28.388759 (+ 1320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:28.389138 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:28.389140 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:28.389141 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:28.389217 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:28.389440 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:28.389443 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:28.389444 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:28.389658 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:28.389764 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:28.389956 (+ 192us) server_negotiation.cc:299] Negotiation successful
0812 09:45:28.390055 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260812 09:45:28.390694 18865 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:40604
I20260812 09:45:28.391567 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 18965
I20260812 09:45:28.405977 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 19176
I20260812 09:45:28.420161 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 19400
I20260812 09:45:28.430660 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 18817
2026-08-12T09:45:28Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2423 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786527905902985-13708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 09:45:28 dist-test-slave-jq4t krb5kdc[19626](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 09:45:28 dist-test-slave-jq4t krb5kdc[19626](info): set up 2 sockets
Aug 12 09:45:28 dist-test-slave-jq4t krb5kdc[19626](info): commencing operation
krb5kdc: starting...
W20260812 09:45:29.743173 13708 mini_kdc.cc:121] Time spent starting KDC: real 1.282s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260812 09:45:29.971853 13708 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.229s user 0.000s sys 0.002s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260812 09:45:30.174223 13708 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.202s user 0.000s sys 0.002s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260812 09:45:30.309478 13708 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.135s user 0.000s sys 0.002s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786527905902985-13708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786527905902985-13708-0/krb5kdc/test-user.keytab.
W20260812 09:45:30.892750 13708 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.583s user 0.000s sys 0.002s
Aug 12 09:45:30 dist-test-slave-jq4t krb5kdc[19626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527930, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T09:45:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T09:45:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.99.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.99.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.99.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab.
Entry for principal kudu/127.13.99.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab.
I20260812 09:45:30.943620 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786527905902985-13708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786527905902985-13708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786527905902985-13708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.62:45281
--webserver_interface=127.13.99.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab
--principal=kudu/127.13.99.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:45869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.99.62:45281
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:45:31.057368 19646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:45:31.057677 19646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:45:31.057749 19646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:45:31.061723 19646 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 09:45:31.061813 19646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:45:31.061852 19646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:45:31.061890 19646 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 09:45:31.061923 19646 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 09:45:31.067245 19646 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:45869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786527905902985-13708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786527905902985-13708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.99.62:45281
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab
--principal=kudu/127.13.99.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.62:45281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.99.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19646
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786527905902985-13708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:45:31.068336 19646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:45:31.069450 19646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:45:31.078088 19660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:45:31.078214 19646 server_base.cc:1034] running on GCE node
W20260812 09:45:31.078109 19656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:31.078110 19657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:45:31.078886 19646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:45:31.079984 19646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:45:31.081252 19646 hybrid_clock.cc:648] HybridClock initialized: now 1786527931081230 us; error 31 us; skew 500 ppm
Aug 12 09:45:31 dist-test-slave-jq4t krb5kdc[19626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527931, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:45:31.084139 19646 init.cc:377] Logged in from keytab as kudu/127.13.99.62@KRBTEST.COM (short username kudu)
I20260812 09:45:31.085084 19646 webserver.cc:460] Webserver started at http://127.13.99.62:40471/ using document root <none> and password file <none>
I20260812 09:45:31.085650 19646 fs_manager.cc:359] Metadata directory not provided
I20260812 09:45:31.085743 19646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:45:31.085984 19646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:45:31.088012 19646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786527905902985-13708-0/minicluster-data/master-0/data/instance:
uuid: "fc89b699b0884073bf7415e0e3389f59"
format_stamp: "Formatted at 2026-08-12 09:45:31 on dist-test-slave-jq4t"
server_key: "3e3be9fe7017780aaece24800a7f91e1"
server_key_iv: "4e94280b58db11343f1fa271f0bb72d0"
server_key_version: "encryptionkey@0"
I20260812 09:45:31.088707 19646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786527905902985-13708-0/minicluster-data/master-0/wal/instance:
uuid: "fc89b699b0884073bf7415e0e3389f59"
format_stamp: "Formatted at 2026-08-12 09:45:31 on dist-test-slave-jq4t"
server_key: "3e3be9fe7017780aaece24800a7f91e1"
server_key_iv: "4e94280b58db11343f1fa271f0bb72d0"
server_key_version: "encryptionkey@0"
I20260812 09:45:31.092417 19646 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260812 09:45:31.094822 19676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:31.096009 19646 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260812 09:45:31.096153 19646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786527905902985-13708-0/minicluster-data/master-0/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786527905902985-13708-0/minicluster-data/master-0/wal
uuid: "fc89b699b0884073bf7415e0e3389f59"
format_stamp: "Formatted at 2026-08-12 09:45:31 on dist-test-slave-jq4t"
server_key: "3e3be9fe7017780aaece24800a7f91e1"
server_key_iv: "4e94280b58db11343f1fa271f0bb72d0"
server_key_version: "encryptionkey@0"
I20260812 09:45:31.096261 19646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786527905902985-13708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786527905902985-13708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786527905902985-13708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:45:31.110774 19646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:45:31.113579 19646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:45:31.113791 19646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:45:31.121006 19646 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.62:45281
I20260812 09:45:31.121028 19755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.62:45281 every 8 connection(s)
I20260812 09:45:31.121943 19646 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
I20260812 09:45:31.124677 19756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:45:31.130254 19756 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc89b699b0884073bf7415e0e3389f59: Bootstrap starting.
I20260812 09:45:31.130383 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 19646
I20260812 09:45:31.130502 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786527905902985-13708-0/minicluster-data/master-0/wal/instance
I20260812 09:45:31.130791 13708 external_mini_cluster.cc:1428] Setting key 1411c3d45a3d522084e40eaa2055bbcb
I20260812 09:45:31.133034 19756 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc89b699b0884073bf7415e0e3389f59: Neither blocks nor log segments found. Creating new log.
I20260812 09:45:31.133772 19756 log.cc:824] T 00000000000000000000000000000000 P fc89b699b0884073bf7415e0e3389f59: Log is configured to *not* fsync() on all Append() calls
I20260812 09:45:31.135617 19756 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc89b699b0884073bf7415e0e3389f59: No bootstrap required, opened a new log
I20260812 09:45:31.139396 19756 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc89b699b0884073bf7415e0e3389f59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc89b699b0884073bf7415e0e3389f59" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 45281 } }
I20260812 09:45:31.139679 19756 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc89b699b0884073bf7415e0e3389f59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:45:31.139760 19756 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc89b699b0884073bf7415e0e3389f59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc89b699b0884073bf7415e0e3389f59, State: Initialized, Role: FOLLOWER
Aug 12 09:45:31 dist-test-slave-jq4t krb5kdc[19626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527930, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
I20260812 09:45:31.140286 19756 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc89b699b0884073bf7415e0e3389f59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc89b699b0884073bf7415e0e3389f59" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 45281 } }
I20260812 09:45:31.140491 19756 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc89b699b0884073bf7415e0e3389f59 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 09:45:31.140573 19756 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc89b699b0884073bf7415e0e3389f59 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 09:45:31.140684 19756 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc89b699b0884073bf7415e0e3389f59 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:45:31.141968 19756 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc89b699b0884073bf7415e0e3389f59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc89b699b0884073bf7415e0e3389f59" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 45281 } }
I20260812 09:45:31.142364 19756 leader_election.cc:302] T 00000000000000000000000000000000 P fc89b699b0884073bf7415e0e3389f59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc89b699b0884073bf7415e0e3389f59; no voters:
I20260812 09:45:31.142722 19756 leader_election.cc:288] T 00000000000000000000000000000000 P fc89b699b0884073bf7415e0e3389f59 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 09:45:31.142848 19761 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc89b699b0884073bf7415e0e3389f59 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 09:45:31.143136 19761 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc89b699b0884073bf7415e0e3389f59 [term 1 LEADER]: Becoming Leader. State: Replica: fc89b699b0884073bf7415e0e3389f59, State: Running, Role: LEADER
I20260812 09:45:31.143522 19761 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc89b699b0884073bf7415e0e3389f59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc89b699b0884073bf7415e0e3389f59" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 45281 } }
I20260812 09:45:31.144233 19756 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc89b699b0884073bf7415e0e3389f59 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 09:45:31.144949 19762 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc89b699b0884073bf7415e0e3389f59 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc89b699b0884073bf7415e0e3389f59. Latest consensus state: current_term: 1 leader_uuid: "fc89b699b0884073bf7415e0e3389f59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc89b699b0884073bf7415e0e3389f59" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 45281 } } }
I20260812 09:45:31.145766 19762 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc89b699b0884073bf7415e0e3389f59 [sys.catalog]: This master's current role is: LEADER
I20260812 09:45:31.145641 19763 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc89b699b0884073bf7415e0e3389f59 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc89b699b0884073bf7415e0e3389f59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc89b699b0884073bf7415e0e3389f59" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 45281 } } }
I20260812 09:45:31.146148 19763 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc89b699b0884073bf7415e0e3389f59 [sys.catalog]: This master's current role is: LEADER
I20260812 09:45:31.146502 19791 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 09:45:31.146911 19759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:31.132233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41608 (local address 127.13.99.62:45281)
0812 09:45:31.132707 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:31.132722 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:31.132757 (+ 35us) server_negotiation.cc:407] Connection header received
0812 09:45:31.133462 (+ 705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:31.133482 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:31.133808 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:31.134178 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:31.135015 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:31.136065 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:31.137001 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:31.137521 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:31.137721 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:31.140647 (+ 2926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:31.140654 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:31.140670 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:31.140707 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:31.142846 (+ 2139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:31.143409 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:31.143415 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:31.143421 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:31.143574 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:31.143927 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:31.143932 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:31.143933 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:31.144511 (+ 578us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:31.144712 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:31.145410 (+ 698us) server_negotiation.cc:299] Negotiation successful
0812 09:45:31.145743 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":162,"threads_started":1}
I20260812 09:45:31.150539 19791 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 09:45:31.156023 19791 catalog_manager.cc:1347] Generated new cluster ID: 542605d3dba8477dacb4fa9fb4c4c773
I20260812 09:45:31.156118 19791 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 09:45:31.162832 19791 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 09:45:31.163681 19791 catalog_manager.cc:1504] Loading token signing keys...
I20260812 09:45:31.168630 19791 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc89b699b0884073bf7415e0e3389f59: Generated new TSK 0
I20260812 09:45:31.169261 19791 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.99.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.99.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.99.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.1.keytab.
Entry for principal kudu/127.13.99.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.1.keytab.
I20260812 09:45:31.212392 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786527905902985-13708-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786527905902985-13708-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786527905902985-13708-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786527905902985-13708-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.99.1:0
--local_ip_for_outbound_sockets=127.13.99.1
--webserver_interface=127.13.99.1
--webserver_port=0
--tserver_master_addrs=127.13.99.62:45281
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.1.keytab
--principal=kudu/127.13.99.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:45869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:45:31.329211 19805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:45:31.329483 19805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:45:31.329526 19805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:45:31.333606 19805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:45:31.333664 19805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:45:31.333721 19805 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.99.1
I20260812 09:45:31.338940 19805 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:45869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786527905902985-13708-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786527905902985-13708-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.1.keytab
--principal=kudu/127.13.99.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.99.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786527905902985-13708-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.99.1
--webserver_port=0
--tserver_master_addrs=127.13.99.62:45281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19805
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.99.1
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786527905902985-13708-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:45:31.339890 19805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:45:31.340970 19805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:45:31.350929 19816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:31.351024 19814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:31.350916 19818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:45:31.351213 19805 server_base.cc:1034] running on GCE node
I20260812 09:45:31.351751 19805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:45:31.352411 19805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:45:31.353596 19805 hybrid_clock.cc:648] HybridClock initialized: now 1786527931353557 us; error 48 us; skew 500 ppm
Aug 12 09:45:31 dist-test-slave-jq4t krb5kdc[19626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527931, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:45:31.356408 19805 init.cc:377] Logged in from keytab as kudu/127.13.99.1@KRBTEST.COM (short username kudu)
I20260812 09:45:31.357313 19805 webserver.cc:460] Webserver started at http://127.13.99.1:35577/ using document root <none> and password file <none>
I20260812 09:45:31.357895 19805 fs_manager.cc:359] Metadata directory not provided
I20260812 09:45:31.358032 19805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:45:31.358278 19805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:45:31.360451 19805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786527905902985-13708-0/minicluster-data/ts-0/data/instance:
uuid: "7a0a92419f75492c8a57a6478ab15971"
format_stamp: "Formatted at 2026-08-12 09:45:31 on dist-test-slave-jq4t"
server_key: "c7c141f767042f3e16d38a30de54d2bf"
server_key_iv: "d4d386965b97d3fada0c295cbceb5cda"
server_key_version: "encryptionkey@0"
I20260812 09:45:31.361156 19805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786527905902985-13708-0/minicluster-data/ts-0/wal/instance:
uuid: "7a0a92419f75492c8a57a6478ab15971"
format_stamp: "Formatted at 2026-08-12 09:45:31 on dist-test-slave-jq4t"
server_key: "c7c141f767042f3e16d38a30de54d2bf"
server_key_iv: "d4d386965b97d3fada0c295cbceb5cda"
server_key_version: "encryptionkey@0"
I20260812 09:45:31.364863 19805 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 09:45:31.367441 19828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:31.368762 19805 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 09:45:31.368888 19805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786527905902985-13708-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786527905902985-13708-0/minicluster-data/ts-0/wal
uuid: "7a0a92419f75492c8a57a6478ab15971"
format_stamp: "Formatted at 2026-08-12 09:45:31 on dist-test-slave-jq4t"
server_key: "c7c141f767042f3e16d38a30de54d2bf"
server_key_iv: "d4d386965b97d3fada0c295cbceb5cda"
server_key_version: "encryptionkey@0"
I20260812 09:45:31.369004 19805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786527905902985-13708-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786527905902985-13708-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786527905902985-13708-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:45:31.390888 19805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:45:31.394130 19805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:45:31.394341 19805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:45:31.394944 19805 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 09:45:31.395789 19805 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:45:31.395862 19805 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:31.395936 19805 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:45:31.395998 19805 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:31.405387 19805 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.1:42899
I20260812 09:45:31.405419 20023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.1:42899 every 8 connection(s)
I20260812 09:45:31.406287 19805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786527905902985-13708-0/minicluster-data/ts-0/data/info.pb
I20260812 09:45:31.409423 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 19805
I20260812 09:45:31.409535 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786527905902985-13708-0/minicluster-data/ts-0/wal/instance
I20260812 09:45:31.409848 13708 external_mini_cluster.cc:1428] Setting key edeb6bdd4d2e05143cf9a01af47ef895
Aug 12 09:45:31 dist-test-slave-jq4t krb5kdc[19626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527931, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.1@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.99.2@KRBTEST.COM; defaulting to no policy
I20260812 09:45:31.420975 19759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:31.408044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.1:51079 (local address 127.13.99.62:45281)
0812 09:45:31.408218 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:31.408223 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:31.409095 (+ 872us) server_negotiation.cc:407] Connection header received
0812 09:45:31.410734 (+ 1639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:31.410737 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:31.410780 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:31.410865 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:31.412234 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:31.412842 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:31.414018 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:31.414262 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:31.414380 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:31.416936 (+ 2556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:31.416944 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:31.416946 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:31.416971 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:31.418469 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:31.419012 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:31.419016 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:31.419018 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:31.419110 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:31.419666 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:31.419677 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:31.419678 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:31.419875 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:31.419966 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:31.420718 (+ 752us) server_negotiation.cc:299] Negotiation successful
0812 09:45:31.420847 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260812 09:45:31.421880 20028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:31.408441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:45281 (local address 127.13.99.1:51079)
0812 09:45:31.408940 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:31.408983 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:31.410497 (+ 1514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:31.411062 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:31.411071 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:31.411595 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:45:31.412073 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:31.412087 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:31.412975 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:31.412979 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:31.413877 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:31.413887 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:31.414421 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:31.414426 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:31.414612 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:31.415236 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:45:31.415253 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:45:31.416765 (+ 1512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:45:31.418614 (+ 1849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:31.418620 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:31.418634 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:31.418897 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:31.419253 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:31.419257 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:31.419261 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:31.419464 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:31.420022 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:45:31.420030 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:45:31.420383 (+ 353us) client_negotiation.cc:769] Sending connection context
0812 09:45:31.420667 (+ 284us) client_negotiation.cc:240] Negotiation successful
0812 09:45:31.420948 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":115,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.99.2@KRBTEST.COM" created.
I20260812 09:45:31.423278 20026 heartbeater.cc:347] Connected to a master server at 127.13.99.62:45281
I20260812 09:45:31.423504 20026 heartbeater.cc:464] Registering TS with master...
I20260812 09:45:31.423985 20026 heartbeater.cc:511] Master 127.13.99.62:45281 requested a full tablet report, sending...
I20260812 09:45:31.425549 19703 ts_manager.cc:194] Registered new tserver with Master: 7a0a92419f75492c8a57a6478ab15971 (127.13.99.1:42899)
I20260812 09:45:31.426704 19703 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.99.1@KRBTEST.COM'} at 127.13.99.1:51079
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.99.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.2.keytab.
Entry for principal kudu/127.13.99.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.2.keytab.
I20260812 09:45:31.440583 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786527905902985-13708-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786527905902985-13708-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786527905902985-13708-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786527905902985-13708-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.99.2:0
--local_ip_for_outbound_sockets=127.13.99.2
--webserver_interface=127.13.99.2
--webserver_port=0
--tserver_master_addrs=127.13.99.62:45281
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.2.keytab
--principal=kudu/127.13.99.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:45869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:45:31.557977 20032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:45:31.558229 20032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:45:31.558272 20032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:45:31.562244 20032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:45:31.562299 20032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:45:31.562355 20032 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.99.2
I20260812 09:45:31.567870 20032 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:45869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786527905902985-13708-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786527905902985-13708-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.2.keytab
--principal=kudu/127.13.99.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.99.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786527905902985-13708-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.99.2
--webserver_port=0
--tserver_master_addrs=127.13.99.62:45281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20032
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.99.2
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786527905902985-13708-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:45:31.568943 20032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:45:31.569939 20032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:45:31.579761 20042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:31.579847 20038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:45:31.579828 20032 server_base.cc:1034] running on GCE node
W20260812 09:45:31.579770 20039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:45:31.580410 20032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:45:31.581133 20032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:45:31.582367 20032 hybrid_clock.cc:648] HybridClock initialized: now 1786527931582354 us; error 32 us; skew 500 ppm
Aug 12 09:45:31 dist-test-slave-jq4t krb5kdc[19626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527931, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:45:31.585500 20032 init.cc:377] Logged in from keytab as kudu/127.13.99.2@KRBTEST.COM (short username kudu)
I20260812 09:45:31.586380 20032 webserver.cc:460] Webserver started at http://127.13.99.2:37821/ using document root <none> and password file <none>
I20260812 09:45:31.586978 20032 fs_manager.cc:359] Metadata directory not provided
I20260812 09:45:31.587044 20032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:45:31.587296 20032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:45:31.589427 20032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786527905902985-13708-0/minicluster-data/ts-1/data/instance:
uuid: "68c0a5dc1fe7484ebec1da9ffef0e37d"
format_stamp: "Formatted at 2026-08-12 09:45:31 on dist-test-slave-jq4t"
server_key: "d9723c3d226ab7275a2f033da2dc5be0"
server_key_iv: "4df4a5696813bfe5fe0e581e78eed022"
server_key_version: "encryptionkey@0"
I20260812 09:45:31.590005 20032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786527905902985-13708-0/minicluster-data/ts-1/wal/instance:
uuid: "68c0a5dc1fe7484ebec1da9ffef0e37d"
format_stamp: "Formatted at 2026-08-12 09:45:31 on dist-test-slave-jq4t"
server_key: "d9723c3d226ab7275a2f033da2dc5be0"
server_key_iv: "4df4a5696813bfe5fe0e581e78eed022"
server_key_version: "encryptionkey@0"
I20260812 09:45:31.593786 20032 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 09:45:31.596238 20057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:31.597580 20032 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260812 09:45:31.597707 20032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786527905902985-13708-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786527905902985-13708-0/minicluster-data/ts-1/wal
uuid: "68c0a5dc1fe7484ebec1da9ffef0e37d"
format_stamp: "Formatted at 2026-08-12 09:45:31 on dist-test-slave-jq4t"
server_key: "d9723c3d226ab7275a2f033da2dc5be0"
server_key_iv: "4df4a5696813bfe5fe0e581e78eed022"
server_key_version: "encryptionkey@0"
I20260812 09:45:31.597856 20032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786527905902985-13708-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786527905902985-13708-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786527905902985-13708-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:45:31.611558 20032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:45:31.615042 20032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:45:31.615259 20032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:45:31.615900 20032 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 09:45:31.616833 20032 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:45:31.616882 20032 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:31.616952 20032 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:45:31.616994 20032 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:31.625942 20032 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.2:40333
I20260812 09:45:31.625972 20252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.2:40333 every 8 connection(s)
I20260812 09:45:31.627012 20032 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786527905902985-13708-0/minicluster-data/ts-1/data/info.pb
I20260812 09:45:31.627974 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 20032
I20260812 09:45:31.628114 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786527905902985-13708-0/minicluster-data/ts-1/wal/instance
I20260812 09:45:31.628434 13708 external_mini_cluster.cc:1428] Setting key f358161708409d0d7005291788f671ca
Aug 12 09:45:31 dist-test-slave-jq4t krb5kdc[19626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527931, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.2@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.99.3@KRBTEST.COM; defaulting to no policy
I20260812 09:45:31.642329 19759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:31.629093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.2:35603 (local address 127.13.99.62:45281)
0812 09:45:31.629217 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:31.629221 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:31.630058 (+ 837us) server_negotiation.cc:407] Connection header received
0812 09:45:31.631512 (+ 1454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:31.631519 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:31.631584 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:31.631680 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:31.632902 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:31.633742 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:31.634954 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:31.635354 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:31.635496 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:31.638260 (+ 2764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:31.638267 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:31.638272 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:31.638303 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:31.640069 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:31.640633 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:31.640636 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:31.640638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:31.640734 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:31.641198 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:31.641201 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:31.641202 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:31.641391 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:31.641473 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:31.642093 (+ 620us) server_negotiation.cc:299] Negotiation successful
0812 09:45:31.642204 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.99.3@KRBTEST.COM" created.
I20260812 09:45:31.643180 20257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:31.629436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:45281 (local address 127.13.99.2:35603)
0812 09:45:31.629914 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:31.629968 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:31.631287 (+ 1319us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:31.631846 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:31.631853 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:31.632195 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:45:31.632751 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:31.632766 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:31.633887 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:31.633890 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:31.634823 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:31.634834 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:31.635510 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:31.635513 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:31.635676 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:31.636339 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:45:31.636355 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:45:31.638079 (+ 1724us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:45:31.640190 (+ 2111us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:31.640195 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:31.640206 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:31.640531 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:31.640854 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:31.640857 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:31.640889 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:31.641080 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:31.641515 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:45:31.641520 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:45:31.641761 (+ 241us) client_negotiation.cc:769] Sending connection context
0812 09:45:31.641990 (+ 229us) client_negotiation.cc:240] Negotiation successful
0812 09:45:31.642268 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":148,"threads_started":1}
I20260812 09:45:31.644553 20254 heartbeater.cc:347] Connected to a master server at 127.13.99.62:45281
I20260812 09:45:31.644843 20254 heartbeater.cc:464] Registering TS with master...
I20260812 09:45:31.645560 20254 heartbeater.cc:511] Master 127.13.99.62:45281 requested a full tablet report, sending...
I20260812 09:45:31.646921 19703 ts_manager.cc:194] Registered new tserver with Master: 68c0a5dc1fe7484ebec1da9ffef0e37d (127.13.99.2:40333)
I20260812 09:45:31.647837 19703 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.99.2@KRBTEST.COM'} at 127.13.99.2:35603
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.99.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.3.keytab.
Entry for principal kudu/127.13.99.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.3.keytab.
I20260812 09:45:31.660257 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786527905902985-13708-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786527905902985-13708-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786527905902985-13708-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786527905902985-13708-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.99.3:0
--local_ip_for_outbound_sockets=127.13.99.3
--webserver_interface=127.13.99.3
--webserver_port=0
--tserver_master_addrs=127.13.99.62:45281
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.3.keytab
--principal=kudu/127.13.99.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:45869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:45:31.778347 20261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:45:31.778621 20261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:45:31.778662 20261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:45:31.782689 20261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:45:31.782745 20261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:45:31.782802 20261 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.99.3
I20260812 09:45:31.787938 20261 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:45869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786527905902985-13708-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786527905902985-13708-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.3.keytab
--principal=kudu/127.13.99.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.99.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786527905902985-13708-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.99.3
--webserver_port=0
--tserver_master_addrs=127.13.99.62:45281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20261
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.99.3
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786527905902985-13708-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:45:31.788959 20261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:45:31.789939 20261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:45:31.799530 20270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:31.799538 20271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:31.799575 20273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:45:31.799823 20261 server_base.cc:1034] running on GCE node
I20260812 09:45:31.800228 20261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:45:31.800983 20261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:45:31.802170 20261 hybrid_clock.cc:648] HybridClock initialized: now 1786527931802158 us; error 29 us; skew 500 ppm
Aug 12 09:45:31 dist-test-slave-jq4t krb5kdc[19626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527931, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:45:31.805104 20261 init.cc:377] Logged in from keytab as kudu/127.13.99.3@KRBTEST.COM (short username kudu)
I20260812 09:45:31.805984 20261 webserver.cc:460] Webserver started at http://127.13.99.3:36739/ using document root <none> and password file <none>
I20260812 09:45:31.806506 20261 fs_manager.cc:359] Metadata directory not provided
I20260812 09:45:31.806566 20261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:45:31.806802 20261 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:45:31.808897 20261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786527905902985-13708-0/minicluster-data/ts-2/data/instance:
uuid: "fe59a21b10fc4c5695746d1c28586732"
format_stamp: "Formatted at 2026-08-12 09:45:31 on dist-test-slave-jq4t"
server_key: "a003128604fa49740f8a124dce9f8f3d"
server_key_iv: "a27ce381319aaa2cbefa1693029f8337"
server_key_version: "encryptionkey@0"
I20260812 09:45:31.809434 20261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786527905902985-13708-0/minicluster-data/ts-2/wal/instance:
uuid: "fe59a21b10fc4c5695746d1c28586732"
format_stamp: "Formatted at 2026-08-12 09:45:31 on dist-test-slave-jq4t"
server_key: "a003128604fa49740f8a124dce9f8f3d"
server_key_iv: "a27ce381319aaa2cbefa1693029f8337"
server_key_version: "encryptionkey@0"
I20260812 09:45:31.813253 20261 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260812 09:45:31.815675 20283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:31.817183 20261 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 09:45:31.817283 20261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786527905902985-13708-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786527905902985-13708-0/minicluster-data/ts-2/wal
uuid: "fe59a21b10fc4c5695746d1c28586732"
format_stamp: "Formatted at 2026-08-12 09:45:31 on dist-test-slave-jq4t"
server_key: "a003128604fa49740f8a124dce9f8f3d"
server_key_iv: "a27ce381319aaa2cbefa1693029f8337"
server_key_version: "encryptionkey@0"
I20260812 09:45:31.817368 20261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786527905902985-13708-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786527905902985-13708-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786527905902985-13708-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:45:31.838294 20261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:45:31.841233 20261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:45:31.841382 20261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:45:31.841909 20261 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 09:45:31.842696 20261 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:45:31.842744 20261 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:31.842784 20261 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:45:31.842803 20261 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:31.851840 20261 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.3:40807
I20260812 09:45:31.851868 20463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.3:40807 every 8 connection(s)
I20260812 09:45:31.852799 20261 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786527905902985-13708-0/minicluster-data/ts-2/data/info.pb
I20260812 09:45:31.857642 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 20261
I20260812 09:45:31.857780 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786527905902985-13708-0/minicluster-data/ts-2/wal/instance
I20260812 09:45:31.858103 13708 external_mini_cluster.cc:1428] Setting key 8a2938ac2ed0635e25a03867e4b5a517
Aug 12 09:45:31 dist-test-slave-jq4t krb5kdc[19626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527931, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.3@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
I20260812 09:45:31.866884 19759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:31.854610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.3:60627 (local address 127.13.99.62:45281)
0812 09:45:31.854749 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:31.854760 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:31.855561 (+ 801us) server_negotiation.cc:407] Connection header received
0812 09:45:31.857023 (+ 1462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:31.857026 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:31.857086 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:31.857169 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:31.858745 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:31.859534 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:31.860944 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:31.861162 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:31.861247 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:31.863557 (+ 2310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:31.863560 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:31.863562 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:31.863585 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:31.865087 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:31.865532 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:31.865534 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:31.865535 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:31.865623 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:31.865910 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:31.865912 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:31.865914 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:31.866094 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:31.866221 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:31.866655 (+ 434us) server_negotiation.cc:299] Negotiation successful
0812 09:45:31.866760 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260812 09:45:31.867714 20470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:31.854961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:45281 (local address 127.13.99.3:60627)
0812 09:45:31.855401 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:31.855444 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:31.856812 (+ 1368us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:31.857343 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:31.857372 (+ 29us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:31.857901 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:45:31.858579 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:31.858603 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:31.859667 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:31.859670 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:31.860826 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:31.860837 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:31.861295 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:31.861298 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:31.861438 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:31.861951 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:45:31.861967 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:45:31.863403 (+ 1436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:45:31.865188 (+ 1785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:31.865192 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:31.865203 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:31.865442 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:31.865703 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:31.865705 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:31.865707 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:31.865830 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:31.866222 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:45:31.866227 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:45:31.866441 (+ 214us) client_negotiation.cc:769] Sending connection context
0812 09:45:31.866589 (+ 148us) client_negotiation.cc:240] Negotiation successful
0812 09:45:31.866818 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":120,"threads_started":1}
I20260812 09:45:31.868896 20467 heartbeater.cc:347] Connected to a master server at 127.13.99.62:45281
I20260812 09:45:31.869149 20467 heartbeater.cc:464] Registering TS with master...
I20260812 09:45:31.869630 20467 heartbeater.cc:511] Master 127.13.99.62:45281 requested a full tablet report, sending...
I20260812 09:45:31.870754 19703 ts_manager.cc:194] Registered new tserver with Master: fe59a21b10fc4c5695746d1c28586732 (127.13.99.3:40807)
I20260812 09:45:31.871369 19703 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.99.3@KRBTEST.COM'} at 127.13.99.3:60627
I20260812 09:45:31.871904 13708 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 09:45:31 dist-test-slave-jq4t krb5kdc[19626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527931, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 09:45:31 dist-test-slave-jq4t krb5kdc[19626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527931, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.99.1@KRBTEST.COM
I20260812 09:45:31.899626 20477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:31.889948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43758 (local address 127.13.99.1:42899)
0812 09:45:31.890270 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:31.890285 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:31.890310 (+ 25us) server_negotiation.cc:407] Connection header received
0812 09:45:31.890452 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:31.890458 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:31.890664 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:31.890788 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:31.891499 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:31.892332 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:31.893213 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:31.893579 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:31.893665 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:31.896050 (+ 2385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:31.896059 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:31.896072 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:31.896096 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:31.897695 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:31.898148 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:31.898154 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:31.898155 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:31.898246 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:31.898506 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:31.898508 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:31.898509 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:31.898751 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:31.898927 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:31.899191 (+ 264us) server_negotiation.cc:299] Negotiation successful
0812 09:45:31.899386 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":105,"threads_started":1}
Aug 12 09:45:31 dist-test-slave-jq4t krb5kdc[19626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527931, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
I20260812 09:45:31.911123 19759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:31.902740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41610 (local address 127.13.99.62:45281)
0812 09:45:31.902877 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:31.902883 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:31.902964 (+ 81us) server_negotiation.cc:407] Connection header received
0812 09:45:31.903128 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:31.903131 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:31.903176 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:31.903261 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:31.903871 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:31.904399 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:31.905285 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:31.905514 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:31.905623 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:31.908125 (+ 2502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:31.908129 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:31.908131 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:31.908155 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:31.909420 (+ 1265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:31.909838 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:31.909846 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:31.909848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:31.909925 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:31.910203 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:31.910208 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:31.910209 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:31.910438 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:31.910576 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:31.910837 (+ 261us) server_negotiation.cc:299] Negotiation successful
0812 09:45:31.910962 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260812 09:45:31.911545 19702 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41610
I20260812 09:45:31.912143 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 19805
I20260812 09:45:31.939286 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 20032
I20260812 09:45:31.958154 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 20261
I20260812 09:45:31.970641 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 19646
2026-08-12T09:45:31Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3539 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 09:45:32 dist-test-slave-jq4t krb5kdc[20483](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 09:45:32 dist-test-slave-jq4t krb5kdc[20483](info): set up 2 sockets
Aug 12 09:45:32 dist-test-slave-jq4t krb5kdc[20483](info): commencing operation
krb5kdc: starting...
W20260812 09:45:33.247225 13708 mini_kdc.cc:121] Time spent starting KDC: real 1.247s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/krb5kdc/test-user.keytab.
Aug 12 09:45:33 dist-test-slave-jq4t krb5kdc[20483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527933, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T09:45:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T09:45:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.99.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.99.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.99.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab.
Entry for principal kudu/127.13.99.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab.
I20260812 09:45:33.376101 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.62:43963
--webserver_interface=127.13.99.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab
--principal=kudu/127.13.99.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:42793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.99.62:43963
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:45:33.492393 20507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:45:33.492693 20507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:45:33.492784 20507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:45:33.496781 20507 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 09:45:33.496834 20507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:45:33.496853 20507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:45:33.496871 20507 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 09:45:33.496886 20507 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 09:45:33.502224 20507 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:42793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.99.62:43963
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab
--principal=kudu/127.13.99.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.62:43963
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.99.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:45:33.503268 20507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:45:33.504253 20507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 09:45:33.512801 20507 server_base.cc:1034] running on GCE node
W20260812 09:45:33.512750 20518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:33.512758 20515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:33.512750 20520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:45:33.513616 20507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:45:33.514683 20507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:45:33.515841 20507 hybrid_clock.cc:648] HybridClock initialized: now 1786527933515821 us; error 33 us; skew 500 ppm
Aug 12 09:45:33 dist-test-slave-jq4t krb5kdc[20483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527933, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:45:33.521626 20507 init.cc:377] Logged in from keytab as kudu/127.13.99.62@KRBTEST.COM (short username kudu)
I20260812 09:45:33.522420 20507 fs_manager.cc:359] Metadata directory not provided
I20260812 09:45:33.522488 20507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:45:33.522723 20507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:45:33.525002 20507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/minicluster-data/master-0/data/instance:
uuid: "761849dbb1644dd9a894f70db486b1b1"
format_stamp: "Formatted at 2026-08-12 09:45:33 on dist-test-slave-jq4t"
server_key: "145ab0b8604d12238f2e4eca8df33310"
server_key_iv: "ea6c2dc4a7369e7e16b1a10b0293c372"
server_key_version: "encryptionkey@0"
I20260812 09:45:33.525592 20507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/minicluster-data/master-0/wal/instance:
uuid: "761849dbb1644dd9a894f70db486b1b1"
format_stamp: "Formatted at 2026-08-12 09:45:33 on dist-test-slave-jq4t"
server_key: "145ab0b8604d12238f2e4eca8df33310"
server_key_iv: "ea6c2dc4a7369e7e16b1a10b0293c372"
server_key_version: "encryptionkey@0"
I20260812 09:45:33.529578 20507 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 09:45:33.532179 20529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:33.533589 20507 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 09:45:33.533742 20507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/minicluster-data/master-0/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/minicluster-data/master-0/wal
uuid: "761849dbb1644dd9a894f70db486b1b1"
format_stamp: "Formatted at 2026-08-12 09:45:33 on dist-test-slave-jq4t"
server_key: "145ab0b8604d12238f2e4eca8df33310"
server_key_iv: "ea6c2dc4a7369e7e16b1a10b0293c372"
server_key_version: "encryptionkey@0"
I20260812 09:45:33.533841 20507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:45:33.548054 20507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:45:33.551234 20507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:45:33.551549 20507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:45:33.558945 20507 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.62:43963
I20260812 09:45:33.559226 20617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.62:43963 every 8 connection(s)
I20260812 09:45:33.559666 20507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
I20260812 09:45:33.562561 20618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:45:33.563577 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 20507
I20260812 09:45:33.563704 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/minicluster-data/master-0/wal/instance
I20260812 09:45:33.564044 13708 external_mini_cluster.cc:1428] Setting key 3e709a924a673809a50464e0a7d9193a
I20260812 09:45:33.569417 20618 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 761849dbb1644dd9a894f70db486b1b1: Bootstrap starting.
I20260812 09:45:33.572186 20618 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 761849dbb1644dd9a894f70db486b1b1: Neither blocks nor log segments found. Creating new log.
Aug 12 09:45:33 dist-test-slave-jq4t krb5kdc[20483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527933, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
I20260812 09:45:33.573087 20618 log.cc:824] T 00000000000000000000000000000000 P 761849dbb1644dd9a894f70db486b1b1: Log is configured to *not* fsync() on all Append() calls
I20260812 09:45:33.574975 20618 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 761849dbb1644dd9a894f70db486b1b1: No bootstrap required, opened a new log
I20260812 09:45:33.578306 20625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:33.565465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45074 (local address 127.13.99.62:43963)
0812 09:45:33.565888 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:33.565903 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:33.565938 (+ 35us) server_negotiation.cc:407] Connection header received
0812 09:45:33.566586 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:33.566602 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:33.566917 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:33.567237 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:33.568151 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:33.569207 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:33.570355 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:33.570824 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:33.570944 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:33.573705 (+ 2761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:33.573711 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:33.573724 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:33.573752 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:33.575531 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:33.575987 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:33.575992 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:33.575997 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:33.576105 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:33.576396 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:33.576400 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:33.576402 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:33.576863 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:33.577085 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:33.577329 (+ 244us) server_negotiation.cc:299] Negotiation successful
0812 09:45:33.577609 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":125,"threads_started":1}
I20260812 09:45:33.578739 20618 raft_consensus.cc:348] T 00000000000000000000000000000000 P 761849dbb1644dd9a894f70db486b1b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "761849dbb1644dd9a894f70db486b1b1" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 43963 } }
I20260812 09:45:33.578991 20618 raft_consensus.cc:374] T 00000000000000000000000000000000 P 761849dbb1644dd9a894f70db486b1b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:45:33.579087 20618 raft_consensus.cc:729] T 00000000000000000000000000000000 P 761849dbb1644dd9a894f70db486b1b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 761849dbb1644dd9a894f70db486b1b1, State: Initialized, Role: FOLLOWER
I20260812 09:45:33.579622 20618 consensus_queue.cc:260] T 00000000000000000000000000000000 P 761849dbb1644dd9a894f70db486b1b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "761849dbb1644dd9a894f70db486b1b1" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 43963 } }
I20260812 09:45:33.579790 20618 raft_consensus.cc:388] T 00000000000000000000000000000000 P 761849dbb1644dd9a894f70db486b1b1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 09:45:33.579882 20618 raft_consensus.cc:482] T 00000000000000000000000000000000 P 761849dbb1644dd9a894f70db486b1b1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 09:45:33.579983 20618 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 761849dbb1644dd9a894f70db486b1b1 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:45:33.581121 20618 raft_consensus.cc:504] T 00000000000000000000000000000000 P 761849dbb1644dd9a894f70db486b1b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "761849dbb1644dd9a894f70db486b1b1" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 43963 } }
I20260812 09:45:33.581441 20618 leader_election.cc:302] T 00000000000000000000000000000000 P 761849dbb1644dd9a894f70db486b1b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 761849dbb1644dd9a894f70db486b1b1; no voters:
I20260812 09:45:33.581696 20618 leader_election.cc:288] T 00000000000000000000000000000000 P 761849dbb1644dd9a894f70db486b1b1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 09:45:33.581827 20627 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 761849dbb1644dd9a894f70db486b1b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 09:45:33.582055 20627 raft_consensus.cc:686] T 00000000000000000000000000000000 P 761849dbb1644dd9a894f70db486b1b1 [term 1 LEADER]: Becoming Leader. State: Replica: 761849dbb1644dd9a894f70db486b1b1, State: Running, Role: LEADER
I20260812 09:45:33.582453 20627 consensus_queue.cc:237] T 00000000000000000000000000000000 P 761849dbb1644dd9a894f70db486b1b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "761849dbb1644dd9a894f70db486b1b1" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 43963 } }
I20260812 09:45:33.582985 20618 sys_catalog.cc:564] T 00000000000000000000000000000000 P 761849dbb1644dd9a894f70db486b1b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 09:45:33.584441 20630 sys_catalog.cc:455] T 00000000000000000000000000000000 P 761849dbb1644dd9a894f70db486b1b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 761849dbb1644dd9a894f70db486b1b1. Latest consensus state: current_term: 1 leader_uuid: "761849dbb1644dd9a894f70db486b1b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "761849dbb1644dd9a894f70db486b1b1" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 43963 } } }
I20260812 09:45:33.584457 20629 sys_catalog.cc:455] T 00000000000000000000000000000000 P 761849dbb1644dd9a894f70db486b1b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "761849dbb1644dd9a894f70db486b1b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "761849dbb1644dd9a894f70db486b1b1" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 43963 } } }
I20260812 09:45:33.584713 20630 sys_catalog.cc:458] T 00000000000000000000000000000000 P 761849dbb1644dd9a894f70db486b1b1 [sys.catalog]: This master's current role is: LEADER
I20260812 09:45:33.584738 20629 sys_catalog.cc:458] T 00000000000000000000000000000000 P 761849dbb1644dd9a894f70db486b1b1 [sys.catalog]: This master's current role is: LEADER
W20260812 09:45:33.587844 20656 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 761849dbb1644dd9a894f70db486b1b1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 09:45:33.587961 20656 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 09:45:33.588060 20650 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 09:45:33.588996 20650 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 09:45:33.594950 20650 catalog_manager.cc:1347] Generated new cluster ID: 13167d6b8758417780312c391802e755
I20260812 09:45:33.595060 20650 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 09:45:33.621876 20650 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 09:45:33.623342 20650 catalog_manager.cc:1504] Loading token signing keys...
I20260812 09:45:33.636914 20650 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 761849dbb1644dd9a894f70db486b1b1: Generated new TSK 0
I20260812 09:45:33.637737 20650 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.99.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.99.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.99.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.1.keytab.
Entry for principal kudu/127.13.99.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.1.keytab.
I20260812 09:45:33.680346 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.99.1:0
--local_ip_for_outbound_sockets=127.13.99.1
--webserver_interface=127.13.99.1
--webserver_port=0
--tserver_master_addrs=127.13.99.62:43963
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.1.keytab
--principal=kudu/127.13.99.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:42793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:45:33.797160 20662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:45:33.797395 20662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:45:33.797438 20662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:45:33.801453 20662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:45:33.801502 20662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:45:33.801559 20662 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.99.1
I20260812 09:45:33.806735 20662 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:42793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.1.keytab
--principal=kudu/127.13.99.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.99.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.99.1
--webserver_port=0
--tserver_master_addrs=127.13.99.62:43963
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20662
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.99.1
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:45:33.807667 20662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:45:33.808725 20662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:45:33.818717 20670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:33.818746 20671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:33.818768 20676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:45:33.819090 20662 server_base.cc:1034] running on GCE node
I20260812 09:45:33.819567 20662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:45:33.820242 20662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:45:33.821442 20662 hybrid_clock.cc:648] HybridClock initialized: now 1786527933821425 us; error 77 us; skew 500 ppm
Aug 12 09:45:33 dist-test-slave-jq4t krb5kdc[20483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527933, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:45:33.824293 20662 init.cc:377] Logged in from keytab as kudu/127.13.99.1@KRBTEST.COM (short username kudu)
I20260812 09:45:33.825114 20662 fs_manager.cc:359] Metadata directory not provided
I20260812 09:45:33.825210 20662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:45:33.825433 20662 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:45:33.827526 20662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/minicluster-data/ts-0/data/instance:
uuid: "d694b8f9ab224e7ea52d5303781494c5"
format_stamp: "Formatted at 2026-08-12 09:45:33 on dist-test-slave-jq4t"
server_key: "6cf10ac9536d0675c810e430ebe37a08"
server_key_iv: "d222368db150babee795424b475a2797"
server_key_version: "encryptionkey@0"
I20260812 09:45:33.828102 20662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/minicluster-data/ts-0/wal/instance:
uuid: "d694b8f9ab224e7ea52d5303781494c5"
format_stamp: "Formatted at 2026-08-12 09:45:33 on dist-test-slave-jq4t"
server_key: "6cf10ac9536d0675c810e430ebe37a08"
server_key_iv: "d222368db150babee795424b475a2797"
server_key_version: "encryptionkey@0"
I20260812 09:45:33.831779 20662 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260812 09:45:33.834232 20682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:33.835603 20662 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260812 09:45:33.835747 20662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/minicluster-data/ts-0/wal
uuid: "d694b8f9ab224e7ea52d5303781494c5"
format_stamp: "Formatted at 2026-08-12 09:45:33 on dist-test-slave-jq4t"
server_key: "6cf10ac9536d0675c810e430ebe37a08"
server_key_iv: "d222368db150babee795424b475a2797"
server_key_version: "encryptionkey@0"
I20260812 09:45:33.835858 20662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:45:33.874006 20662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:45:33.877182 20662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:45:33.877393 20662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:45:33.878012 20662 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 09:45:33.878983 20662 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:45:33.879055 20662 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:33.879132 20662 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:45:33.879175 20662 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:33.888319 20662 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.1:37255
I20260812 09:45:33.888358 20854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.1:37255 every 8 connection(s)
I20260812 09:45:33.889019 20662 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/minicluster-data/ts-0/data/info.pb
Aug 12 09:45:33 dist-test-slave-jq4t krb5kdc[20483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527933, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.1@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
I20260812 09:45:33.899027 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 20662
I20260812 09:45:33.899173 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/minicluster-data/ts-0/wal/instance
I20260812 09:45:33.899528 13708 external_mini_cluster.cc:1428] Setting key 46db20e379472c5fe23ace1ac1c95022
I20260812 09:45:33.902164 20625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:33.890775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.1:35179 (local address 127.13.99.62:43963)
0812 09:45:33.890953 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:33.890958 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:33.892092 (+ 1134us) server_negotiation.cc:407] Connection header received
0812 09:45:33.892990 (+ 898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:33.892993 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:33.893039 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:33.893112 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:33.894248 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:33.894780 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:33.895879 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:33.896106 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:33.896192 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:33.898542 (+ 2350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:33.898545 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:33.898548 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:33.898568 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:33.900084 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:33.900633 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:33.900637 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:33.900639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:33.900725 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:33.901120 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:33.901123 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:33.901125 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:33.901302 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:33.901412 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:33.901882 (+ 470us) server_negotiation.cc:299] Negotiation successful
0812 09:45:33.902001 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260812 09:45:33.902895 20857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:33.891157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:43963 (local address 127.13.99.1:35179)
0812 09:45:33.891943 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:33.891985 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:33.892801 (+ 816us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:33.893273 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:33.893279 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:33.893620 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:45:33.894118 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:33.894131 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:33.894903 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:33.894906 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:33.895770 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:33.895779 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:33.896239 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:33.896242 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:33.896376 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:33.897013 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:45:33.897029 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:45:33.898399 (+ 1370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:45:33.900217 (+ 1818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:33.900221 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:33.900232 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:33.900519 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:33.900829 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:33.900834 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:33.900846 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:33.901024 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:33.901410 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:45:33.901417 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:45:33.901647 (+ 230us) client_negotiation.cc:769] Sending connection context
0812 09:45:33.901822 (+ 175us) client_negotiation.cc:240] Negotiation successful
0812 09:45:33.902142 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":111,"threads_started":1}
I20260812 09:45:33.904076 20855 heartbeater.cc:347] Connected to a master server at 127.13.99.62:43963
I20260812 09:45:33.904310 20855 heartbeater.cc:464] Registering TS with master...
I20260812 09:45:33.904795 20855 heartbeater.cc:511] Master 127.13.99.62:43963 requested a full tablet report, sending...
I20260812 09:45:33.906250 20557 ts_manager.cc:194] Registered new tserver with Master: d694b8f9ab224e7ea52d5303781494c5 (127.13.99.1:37255)
I20260812 09:45:33.907325 20557 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.99.1@KRBTEST.COM'} at 127.13.99.1:35179
WARNING: no policy specified for kudu/127.13.99.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.99.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.99.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.2.keytab.
Entry for principal kudu/127.13.99.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.2.keytab.
I20260812 09:45:33.928297 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.99.2:0
--local_ip_for_outbound_sockets=127.13.99.2
--webserver_interface=127.13.99.2
--webserver_port=0
--tserver_master_addrs=127.13.99.62:43963
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.2.keytab
--principal=kudu/127.13.99.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:42793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:45:34.041548 20860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:45:34.041790 20860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:45:34.041836 20860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:45:34.045730 20860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:45:34.045778 20860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:45:34.045837 20860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.99.2
I20260812 09:45:34.050889 20860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:42793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.2.keytab
--principal=kudu/127.13.99.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.99.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.99.2
--webserver_port=0
--tserver_master_addrs=127.13.99.62:43963
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.99.2
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:45:34.051792 20860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:45:34.052750 20860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:45:34.062404 20871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:34.062384 20868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:34.062377 20869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:45:34.062600 20860 server_base.cc:1034] running on GCE node
I20260812 09:45:34.063058 20860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:45:34.063740 20860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:45:34.064926 20860 hybrid_clock.cc:648] HybridClock initialized: now 1786527934064893 us; error 56 us; skew 500 ppm
Aug 12 09:45:34 dist-test-slave-jq4t krb5kdc[20483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527934, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:45:34.067689 20860 init.cc:377] Logged in from keytab as kudu/127.13.99.2@KRBTEST.COM (short username kudu)
I20260812 09:45:34.068562 20860 fs_manager.cc:359] Metadata directory not provided
I20260812 09:45:34.068660 20860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:45:34.068907 20860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:45:34.070988 20860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/minicluster-data/ts-1/data/instance:
uuid: "ab93569b0cbf420da1da9a02c5c9303d"
format_stamp: "Formatted at 2026-08-12 09:45:34 on dist-test-slave-jq4t"
server_key: "424ef37ee70b94b8352e1773779b2f52"
server_key_iv: "d342d09a81828bb3579549962e6d6928"
server_key_version: "encryptionkey@0"
I20260812 09:45:34.071575 20860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/minicluster-data/ts-1/wal/instance:
uuid: "ab93569b0cbf420da1da9a02c5c9303d"
format_stamp: "Formatted at 2026-08-12 09:45:34 on dist-test-slave-jq4t"
server_key: "424ef37ee70b94b8352e1773779b2f52"
server_key_iv: "d342d09a81828bb3579549962e6d6928"
server_key_version: "encryptionkey@0"
I20260812 09:45:34.075565 20860 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 09:45:34.078203 20880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:34.079560 20860 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 09:45:34.079710 20860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/minicluster-data/ts-1/wal
uuid: "ab93569b0cbf420da1da9a02c5c9303d"
format_stamp: "Formatted at 2026-08-12 09:45:34 on dist-test-slave-jq4t"
server_key: "424ef37ee70b94b8352e1773779b2f52"
server_key_iv: "d342d09a81828bb3579549962e6d6928"
server_key_version: "encryptionkey@0"
I20260812 09:45:34.079828 20860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:45:34.097121 20860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:45:34.100050 20860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:45:34.100262 20860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:45:34.100929 20860 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 09:45:34.101773 20860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:45:34.101842 20860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:34.101915 20860 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:45:34.101965 20860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:34.111099 20860 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.2:41419
I20260812 09:45:34.111123 21061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.2:41419 every 8 connection(s)
I20260812 09:45:34.111744 20860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/minicluster-data/ts-1/data/info.pb
I20260812 09:45:34.114708 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 20860
I20260812 09:45:34.114818 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/minicluster-data/ts-1/wal/instance
I20260812 09:45:34.115135 13708 external_mini_cluster.cc:1428] Setting key 6864d954cd21be921f043d595db10578
Aug 12 09:45:34 dist-test-slave-jq4t krb5kdc[20483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527934, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.2@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
I20260812 09:45:34.126730 20625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:34.113533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.2:52215 (local address 127.13.99.62:43963)
0812 09:45:34.113668 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:34.113672 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:34.114894 (+ 1222us) server_negotiation.cc:407] Connection header received
0812 09:45:34.115964 (+ 1070us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:34.115967 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:34.116010 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:34.116072 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:34.117423 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:34.117951 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:34.119569 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:34.119792 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:34.119897 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:34.122664 (+ 2767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:34.122669 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:34.122672 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:34.122704 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:34.124556 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:34.125049 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:34.125056 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:34.125060 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:34.125163 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:34.125515 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:34.125520 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:34.125522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:34.125773 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:34.125887 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:34.126384 (+ 497us) server_negotiation.cc:299] Negotiation successful
0812 09:45:34.126545 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
WARNING: no policy specified for kudu/127.13.99.3@KRBTEST.COM; defaulting to no policy
I20260812 09:45:34.127429 21064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:34.113914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:43963 (local address 127.13.99.2:52215)
0812 09:45:34.114728 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:34.114796 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:34.115745 (+ 949us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:34.116237 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:34.116245 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:34.116788 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:45:34.117274 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:34.117287 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:34.118083 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:34.118086 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:34.119432 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:34.119448 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:34.119951 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:34.119959 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:34.120139 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:34.120761 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:45:34.120777 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:45:34.122488 (+ 1711us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:45:34.124670 (+ 2182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:34.124675 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:34.124686 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:34.124941 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:34.125266 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:34.125268 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:34.125270 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:34.125402 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:34.125902 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:45:34.125907 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:45:34.126128 (+ 221us) client_negotiation.cc:769] Sending connection context
0812 09:45:34.126280 (+ 152us) client_negotiation.cc:240] Negotiation successful
0812 09:45:34.126523 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":116,"threads_started":1}
I20260812 09:45:34.128657 21062 heartbeater.cc:347] Connected to a master server at 127.13.99.62:43963
I20260812 09:45:34.128911 21062 heartbeater.cc:464] Registering TS with master...
I20260812 09:45:34.129420 21062 heartbeater.cc:511] Master 127.13.99.62:43963 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.99.3@KRBTEST.COM" created.
I20260812 09:45:34.130648 20557 ts_manager.cc:194] Registered new tserver with Master: ab93569b0cbf420da1da9a02c5c9303d (127.13.99.2:41419)
I20260812 09:45:34.131318 20557 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.99.2@KRBTEST.COM'} at 127.13.99.2:52215
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.99.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.3.keytab.
Entry for principal kudu/127.13.99.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.3.keytab.
I20260812 09:45:34.145522 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.99.3:0
--local_ip_for_outbound_sockets=127.13.99.3
--webserver_interface=127.13.99.3
--webserver_port=0
--tserver_master_addrs=127.13.99.62:43963
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.3.keytab
--principal=kudu/127.13.99.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:42793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:45:34.260722 21067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:45:34.260987 21067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:45:34.261034 21067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:45:34.265139 21067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:45:34.265198 21067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:45:34.265292 21067 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.99.3
I20260812 09:45:34.270601 21067 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:42793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.3.keytab
--principal=kudu/127.13.99.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.99.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.99.3
--webserver_port=0
--tserver_master_addrs=127.13.99.62:43963
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21067
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.99.3
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:45:34.271705 21067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:45:34.272854 21067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:45:34.282757 21073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:34.282820 21074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:45:34.282927 21067 server_base.cc:1034] running on GCE node
W20260812 09:45:34.282806 21076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:45:34.283468 21067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:45:34.284267 21067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:45:34.285455 21067 hybrid_clock.cc:648] HybridClock initialized: now 1786527934285405 us; error 66 us; skew 500 ppm
Aug 12 09:45:34 dist-test-slave-jq4t krb5kdc[20483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527934, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:45:34.288185 21067 init.cc:377] Logged in from keytab as kudu/127.13.99.3@KRBTEST.COM (short username kudu)
I20260812 09:45:34.288957 21067 fs_manager.cc:359] Metadata directory not provided
I20260812 09:45:34.289055 21067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:45:34.289290 21067 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:45:34.291289 21067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/minicluster-data/ts-2/data/instance:
uuid: "fc730765a72740d0aec85da439bb5545"
format_stamp: "Formatted at 2026-08-12 09:45:34 on dist-test-slave-jq4t"
server_key: "83b6b8239a15910c8812b358530660a3"
server_key_iv: "1a2d026d57e89ffe1b6aa39c985b1e31"
server_key_version: "encryptionkey@0"
I20260812 09:45:34.291888 21067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/minicluster-data/ts-2/wal/instance:
uuid: "fc730765a72740d0aec85da439bb5545"
format_stamp: "Formatted at 2026-08-12 09:45:34 on dist-test-slave-jq4t"
server_key: "83b6b8239a15910c8812b358530660a3"
server_key_iv: "1a2d026d57e89ffe1b6aa39c985b1e31"
server_key_version: "encryptionkey@0"
I20260812 09:45:34.295495 21067 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260812 09:45:34.297838 21086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:34.298978 21067 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260812 09:45:34.299109 21067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/minicluster-data/ts-2/wal
uuid: "fc730765a72740d0aec85da439bb5545"
format_stamp: "Formatted at 2026-08-12 09:45:34 on dist-test-slave-jq4t"
server_key: "83b6b8239a15910c8812b358530660a3"
server_key_iv: "1a2d026d57e89ffe1b6aa39c985b1e31"
server_key_version: "encryptionkey@0"
I20260812 09:45:34.299233 21067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:45:34.318365 21067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:45:34.321641 21067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:45:34.321839 21067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:45:34.322474 21067 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 09:45:34.323323 21067 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:45:34.323401 21067 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:34.323472 21067 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:45:34.323514 21067 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:34.332744 21067 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.3:44285
I20260812 09:45:34.332763 21279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.3:44285 every 8 connection(s)
I20260812 09:45:34.333328 21067 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/minicluster-data/ts-2/data/info.pb
I20260812 09:45:34.342849 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 21067
I20260812 09:45:34.342957 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/minicluster-data/ts-2/wal/instance
I20260812 09:45:34.343277 13708 external_mini_cluster.cc:1428] Setting key a99c9209b03fbb26a2389972792c4a89
Aug 12 09:45:34 dist-test-slave-jq4t krb5kdc[20483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527934, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.3@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
I20260812 09:45:34.347579 20625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:34.335078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.3:46467 (local address 127.13.99.62:43963)
0812 09:45:34.335233 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:34.335238 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:34.336450 (+ 1212us) server_negotiation.cc:407] Connection header received
0812 09:45:34.337552 (+ 1102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:34.337554 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:34.337602 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:34.337679 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:34.339237 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:34.340008 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:34.341409 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:34.341636 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:34.341718 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:34.344283 (+ 2565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:34.344287 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:34.344290 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:34.344323 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:34.345845 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:34.346307 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:34.346309 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:34.346311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:34.346385 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:34.346688 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:34.346690 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:34.346691 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:34.346855 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:34.346944 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:34.347368 (+ 424us) server_negotiation.cc:299] Negotiation successful
0812 09:45:34.347465 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260812 09:45:34.348399 21286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:34.335516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:43963 (local address 127.13.99.3:46467)
0812 09:45:34.336302 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:34.336342 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:34.337356 (+ 1014us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:34.337940 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:34.337949 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:34.338464 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:45:34.339050 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:34.339067 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:34.340136 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:34.340139 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:34.341293 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:34.341303 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:34.341769 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:34.341772 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:34.341922 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:34.342450 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:45:34.342466 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:45:34.344093 (+ 1627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:45:34.345950 (+ 1857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:34.345955 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:34.345966 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:34.346220 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:34.346467 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:34.346469 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:34.346470 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:34.346609 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:34.346944 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:45:34.346949 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:45:34.347170 (+ 221us) client_negotiation.cc:769] Sending connection context
0812 09:45:34.347324 (+ 154us) client_negotiation.cc:240] Negotiation successful
0812 09:45:34.347571 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":108,"threads_started":1}
I20260812 09:45:34.349629 21281 heartbeater.cc:347] Connected to a master server at 127.13.99.62:43963
I20260812 09:45:34.349833 21281 heartbeater.cc:464] Registering TS with master...
I20260812 09:45:34.350246 21281 heartbeater.cc:511] Master 127.13.99.62:43963 requested a full tablet report, sending...
I20260812 09:45:34.351200 20557 ts_manager.cc:194] Registered new tserver with Master: fc730765a72740d0aec85da439bb5545 (127.13.99.3:44285)
I20260812 09:45:34.351733 20557 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.99.3@KRBTEST.COM'} at 127.13.99.3:46467
I20260812 09:45:34.357257 13708 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 09:45:34.366250 20625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:34.359266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45082 (local address 127.13.99.62:43963)
0812 09:45:34.359395 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:34.359406 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:34.359511 (+ 105us) server_negotiation.cc:407] Connection header received
0812 09:45:34.359609 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:34.359612 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:34.359649 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:34.359718 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:34.360418 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:34.361006 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:34.361832 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:34.362038 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:34.362116 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:34.363425 (+ 1309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:34.363428 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:34.363430 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:34.363452 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:34.364813 (+ 1361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:34.365210 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:34.365213 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:34.365214 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:34.365292 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:34.365544 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:34.365547 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:34.365548 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:34.365751 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:34.365883 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:34.366047 (+ 164us) server_negotiation.cc:299] Negotiation successful
0812 09:45:34.366138 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 09:45:34.369223 20557 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45082:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 09:45:34.372295 20557 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 09:45:34.387696 21300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:34.381023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.3:44285 (local address 127.0.0.1:34826)
0812 09:45:34.381528 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:34.381576 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:34.381736 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:34.383360 (+ 1624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:34.383363 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:34.383383 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:45:34.383497 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:34.383503 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:34.385097 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:34.385101 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:34.386316 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:34.386325 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:34.387115 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:34.387119 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:34.387224 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:34.387260 (+ 36us) client_negotiation.cc:769] Sending connection context
0812 09:45:34.387395 (+ 135us) client_negotiation.cc:240] Negotiation successful
0812 09:45:34.387517 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"mutex_wait_us":44,"thread_start_us":80,"threads_started":1}
I20260812 09:45:34.387974 21304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:34.381180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34826 (local address 127.13.99.3:44285)
0812 09:45:34.382877 (+ 1697us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:34.382892 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:34.382919 (+ 27us) server_negotiation.cc:407] Connection header received
0812 09:45:34.383000 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:34.383005 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:34.383189 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:34.383655 (+ 466us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:45:34.383801 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:34.384948 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:34.386441 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:34.386989 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:34.387113 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:34.387262 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:34.387718 (+ 456us) server_negotiation.cc:299] Negotiation successful
0812 09:45:34.387822 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1527,"thread_start_us":157,"threads_started":1}
I20260812 09:45:34.388077 21306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:34.381206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.2:41419 (local address 127.0.0.1:45096)
0812 09:45:34.382117 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:34.382131 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:34.382220 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:34.382736 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:34.382742 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:34.382766 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:45:34.382910 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:34.382917 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:34.384719 (+ 1802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:34.384722 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:34.386343 (+ 1621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:34.386349 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:34.387415 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:34.387418 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:34.387499 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:34.387524 (+ 25us) client_negotiation.cc:769] Sending connection context
0812 09:45:34.387595 (+ 71us) client_negotiation.cc:240] Negotiation successful
0812 09:45:34.387928 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":845,"mutex_wait_us":377,"thread_start_us":102,"threads_started":1}
I20260812 09:45:34.388307 21302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:34.381820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.1:37255 (local address 127.0.0.1:54262)
0812 09:45:34.382462 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:34.382476 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:34.382566 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:34.383735 (+ 1169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:34.383739 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:34.383764 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:45:34.383867 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:34.383874 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:34.385401 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:34.385407 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:34.386336 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:34.386348 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:34.387427 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:34.387431 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:34.387529 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:34.387552 (+ 23us) client_negotiation.cc:769] Sending connection context
0812 09:45:34.387607 (+ 55us) client_negotiation.cc:240] Negotiation successful
0812 09:45:34.387714 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"mutex_wait_us":52,"thread_start_us":109,"threads_started":1}
I20260812 09:45:34.388679 21311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:34.381797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45096 (local address 127.13.99.2:41419)
0812 09:45:34.382175 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:34.382188 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:34.382213 (+ 25us) server_negotiation.cc:407] Connection header received
0812 09:45:34.382339 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:34.382346 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:34.382567 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:34.382710 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:45:34.383020 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:34.384525 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:34.386483 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:34.387282 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:34.387412 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:34.387545 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:34.387638 (+ 93us) server_negotiation.cc:299] Negotiation successful
0812 09:45:34.387724 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":155,"threads_started":1}
I20260812 09:45:34.388948 21314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:34.382434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54262 (local address 127.13.99.1:37255)
0812 09:45:34.383186 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:34.383195 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:34.383220 (+ 25us) server_negotiation.cc:407] Connection header received
0812 09:45:34.383312 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:34.383318 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:34.383554 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:34.384041 (+ 487us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:45:34.384206 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:34.385275 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:34.386483 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:34.387282 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:34.388466 (+ 1184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:34.388632 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:34.388684 (+ 52us) server_negotiation.cc:299] Negotiation successful
0812 09:45:34.388792 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":645,"thread_start_us":94,"threads_started":1}
I20260812 09:45:34.391217 20953 tablet_service.cc:1467] Processing CreateTablet for tablet e1993087690b402bbfb7d59c4acf941f (DEFAULT_TABLE table=test-table [id=af741b660e3d4b2d92639a51c6d89b3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 09:45:34.391184 21174 tablet_service.cc:1467] Processing CreateTablet for tablet e1993087690b402bbfb7d59c4acf941f (DEFAULT_TABLE table=test-table [id=af741b660e3d4b2d92639a51c6d89b3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 09:45:34.391429 20756 tablet_service.cc:1467] Processing CreateTablet for tablet e1993087690b402bbfb7d59c4acf941f (DEFAULT_TABLE table=test-table [id=af741b660e3d4b2d92639a51c6d89b3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 09:45:34.392313 21174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1993087690b402bbfb7d59c4acf941f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:45:34.392309 20756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1993087690b402bbfb7d59c4acf941f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:45:34.392311 20953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1993087690b402bbfb7d59c4acf941f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:45:34.398567 21316 tablet_bootstrap.cc:492] T e1993087690b402bbfb7d59c4acf941f P ab93569b0cbf420da1da9a02c5c9303d: Bootstrap starting.
I20260812 09:45:34.399235 21319 tablet_bootstrap.cc:492] T e1993087690b402bbfb7d59c4acf941f P fc730765a72740d0aec85da439bb5545: Bootstrap starting.
I20260812 09:45:34.400283 21320 tablet_bootstrap.cc:492] T e1993087690b402bbfb7d59c4acf941f P d694b8f9ab224e7ea52d5303781494c5: Bootstrap starting.
I20260812 09:45:34.401304 21319 tablet_bootstrap.cc:654] T e1993087690b402bbfb7d59c4acf941f P fc730765a72740d0aec85da439bb5545: Neither blocks nor log segments found. Creating new log.
I20260812 09:45:34.401655 21316 tablet_bootstrap.cc:654] T e1993087690b402bbfb7d59c4acf941f P ab93569b0cbf420da1da9a02c5c9303d: Neither blocks nor log segments found. Creating new log.
I20260812 09:45:34.402258 21320 tablet_bootstrap.cc:654] T e1993087690b402bbfb7d59c4acf941f P d694b8f9ab224e7ea52d5303781494c5: Neither blocks nor log segments found. Creating new log.
I20260812 09:45:34.402592 21319 log.cc:824] T e1993087690b402bbfb7d59c4acf941f P fc730765a72740d0aec85da439bb5545: Log is configured to *not* fsync() on all Append() calls
I20260812 09:45:34.402592 21316 log.cc:824] T e1993087690b402bbfb7d59c4acf941f P ab93569b0cbf420da1da9a02c5c9303d: Log is configured to *not* fsync() on all Append() calls
I20260812 09:45:34.403417 21320 log.cc:824] T e1993087690b402bbfb7d59c4acf941f P d694b8f9ab224e7ea52d5303781494c5: Log is configured to *not* fsync() on all Append() calls
I20260812 09:45:34.404701 21316 tablet_bootstrap.cc:492] T e1993087690b402bbfb7d59c4acf941f P ab93569b0cbf420da1da9a02c5c9303d: No bootstrap required, opened a new log
I20260812 09:45:34.404733 21319 tablet_bootstrap.cc:492] T e1993087690b402bbfb7d59c4acf941f P fc730765a72740d0aec85da439bb5545: No bootstrap required, opened a new log
I20260812 09:45:34.404910 21316 ts_tablet_manager.cc:1397] T e1993087690b402bbfb7d59c4acf941f P ab93569b0cbf420da1da9a02c5c9303d: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260812 09:45:34.404912 21319 ts_tablet_manager.cc:1397] T e1993087690b402bbfb7d59c4acf941f P fc730765a72740d0aec85da439bb5545: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260812 09:45:34.405391 21320 tablet_bootstrap.cc:492] T e1993087690b402bbfb7d59c4acf941f P d694b8f9ab224e7ea52d5303781494c5: No bootstrap required, opened a new log
I20260812 09:45:34.405566 21320 ts_tablet_manager.cc:1397] T e1993087690b402bbfb7d59c4acf941f P d694b8f9ab224e7ea52d5303781494c5: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260812 09:45:34.407774 21316 raft_consensus.cc:348] T e1993087690b402bbfb7d59c4acf941f P ab93569b0cbf420da1da9a02c5c9303d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc730765a72740d0aec85da439bb5545" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 44285 } } peers { permanent_uuid: "ab93569b0cbf420da1da9a02c5c9303d" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 41419 } } peers { permanent_uuid: "d694b8f9ab224e7ea52d5303781494c5" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 37255 } }
I20260812 09:45:34.408023 21316 raft_consensus.cc:374] T e1993087690b402bbfb7d59c4acf941f P ab93569b0cbf420da1da9a02c5c9303d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:45:34.408079 21316 raft_consensus.cc:729] T e1993087690b402bbfb7d59c4acf941f P ab93569b0cbf420da1da9a02c5c9303d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab93569b0cbf420da1da9a02c5c9303d, State: Initialized, Role: FOLLOWER
I20260812 09:45:34.408548 21316 consensus_queue.cc:260] T e1993087690b402bbfb7d59c4acf941f P ab93569b0cbf420da1da9a02c5c9303d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc730765a72740d0aec85da439bb5545" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 44285 } } peers { permanent_uuid: "ab93569b0cbf420da1da9a02c5c9303d" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 41419 } } peers { permanent_uuid: "d694b8f9ab224e7ea52d5303781494c5" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 37255 } }
I20260812 09:45:34.408730 21319 raft_consensus.cc:348] T e1993087690b402bbfb7d59c4acf941f P fc730765a72740d0aec85da439bb5545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc730765a72740d0aec85da439bb5545" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 44285 } } peers { permanent_uuid: "ab93569b0cbf420da1da9a02c5c9303d" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 41419 } } peers { permanent_uuid: "d694b8f9ab224e7ea52d5303781494c5" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 37255 } }
I20260812 09:45:34.408994 21319 raft_consensus.cc:374] T e1993087690b402bbfb7d59c4acf941f P fc730765a72740d0aec85da439bb5545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:45:34.409054 21319 raft_consensus.cc:729] T e1993087690b402bbfb7d59c4acf941f P fc730765a72740d0aec85da439bb5545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc730765a72740d0aec85da439bb5545, State: Initialized, Role: FOLLOWER
I20260812 09:45:34.409332 21062 heartbeater.cc:503] Master 127.13.99.62:43963 was elected leader, sending a full tablet report...
I20260812 09:45:34.409178 21320 raft_consensus.cc:348] T e1993087690b402bbfb7d59c4acf941f P d694b8f9ab224e7ea52d5303781494c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc730765a72740d0aec85da439bb5545" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 44285 } } peers { permanent_uuid: "ab93569b0cbf420da1da9a02c5c9303d" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 41419 } } peers { permanent_uuid: "d694b8f9ab224e7ea52d5303781494c5" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 37255 } }
I20260812 09:45:34.409411 21320 raft_consensus.cc:374] T e1993087690b402bbfb7d59c4acf941f P d694b8f9ab224e7ea52d5303781494c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:45:34.409464 21320 raft_consensus.cc:729] T e1993087690b402bbfb7d59c4acf941f P d694b8f9ab224e7ea52d5303781494c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d694b8f9ab224e7ea52d5303781494c5, State: Initialized, Role: FOLLOWER
I20260812 09:45:34.409577 21316 ts_tablet_manager.cc:1428] T e1993087690b402bbfb7d59c4acf941f P ab93569b0cbf420da1da9a02c5c9303d: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260812 09:45:34.409888 21320 consensus_queue.cc:260] T e1993087690b402bbfb7d59c4acf941f P d694b8f9ab224e7ea52d5303781494c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc730765a72740d0aec85da439bb5545" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 44285 } } peers { permanent_uuid: "ab93569b0cbf420da1da9a02c5c9303d" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 41419 } } peers { permanent_uuid: "d694b8f9ab224e7ea52d5303781494c5" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 37255 } }
I20260812 09:45:34.410257 21319 consensus_queue.cc:260] T e1993087690b402bbfb7d59c4acf941f P fc730765a72740d0aec85da439bb5545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc730765a72740d0aec85da439bb5545" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 44285 } } peers { permanent_uuid: "ab93569b0cbf420da1da9a02c5c9303d" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 41419 } } peers { permanent_uuid: "d694b8f9ab224e7ea52d5303781494c5" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 37255 } }
I20260812 09:45:34.410629 20855 heartbeater.cc:503] Master 127.13.99.62:43963 was elected leader, sending a full tablet report...
I20260812 09:45:34.410936 21320 ts_tablet_manager.cc:1428] T e1993087690b402bbfb7d59c4acf941f P d694b8f9ab224e7ea52d5303781494c5: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260812 09:45:34.411230 21319 ts_tablet_manager.cc:1428] T e1993087690b402bbfb7d59c4acf941f P fc730765a72740d0aec85da439bb5545: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260812 09:45:34.411607 21281 heartbeater.cc:503] Master 127.13.99.62:43963 was elected leader, sending a full tablet report...
I20260812 09:45:34.439527 21326 raft_consensus.cc:482] T e1993087690b402bbfb7d59c4acf941f P ab93569b0cbf420da1da9a02c5c9303d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 09:45:34.439739 21326 raft_consensus.cc:504] T e1993087690b402bbfb7d59c4acf941f P ab93569b0cbf420da1da9a02c5c9303d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc730765a72740d0aec85da439bb5545" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 44285 } } peers { permanent_uuid: "ab93569b0cbf420da1da9a02c5c9303d" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 41419 } } peers { permanent_uuid: "d694b8f9ab224e7ea52d5303781494c5" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 37255 } }
I20260812 09:45:34.441175 21326 leader_election.cc:288] T e1993087690b402bbfb7d59c4acf941f P ab93569b0cbf420da1da9a02c5c9303d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc730765a72740d0aec85da439bb5545 (127.13.99.3:44285), d694b8f9ab224e7ea52d5303781494c5 (127.13.99.1:37255)
I20260812 09:45:34.444900 21064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:34.441383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.3:44285 (local address 127.13.99.2:58237)
0812 09:45:34.441564 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:34.441584 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:34.441683 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:34.442030 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:34.442036 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:34.442067 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:45:34.442167 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:34.442172 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:34.442946 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:34.442949 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:34.443738 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:34.443747 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:34.444434 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:34.444437 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:34.444579 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:34.444603 (+ 24us) client_negotiation.cc:769] Sending connection context
0812 09:45:34.444696 (+ 93us) client_negotiation.cc:240] Negotiation successful
0812 09:45:34.444766 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84}
I20260812 09:45:34.444906 21304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:34.441543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.2:58237 (local address 127.13.99.3:44285)
0812 09:45:34.441665 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:34.441669 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:34.441683 (+ 14us) server_negotiation.cc:407] Connection header received
0812 09:45:34.441828 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:34.441832 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:34.441896 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:34.441982 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:45:34.442292 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:34.442800 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:34.443876 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:34.444298 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:34.444430 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:34.444541 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:34.444715 (+ 174us) server_negotiation.cc:299] Negotiation successful
0812 09:45:34.444792 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260812 09:45:34.445432 21329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:34.441383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.1:37255 (local address 127.13.99.2:60571)
0812 09:45:34.442109 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:34.442123 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:34.442247 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:34.442487 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:34.442490 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:34.442506 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:45:34.442634 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:34.442640 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:34.443385 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:34.443388 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:34.444180 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:34.444188 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:34.444949 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:34.444953 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:34.445064 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:34.445091 (+ 27us) client_negotiation.cc:769] Sending connection context
0812 09:45:34.445210 (+ 119us) client_negotiation.cc:240] Negotiation successful
0812 09:45:34.445280 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":649,"mutex_wait_us":322,"thread_start_us":137,"threads_started":1}
I20260812 09:45:34.445456 21314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:34.441504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.2:60571 (local address 127.13.99.1:37255)
0812 09:45:34.441634 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:34.441640 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:34.442214 (+ 574us) server_negotiation.cc:407] Connection header received
0812 09:45:34.442334 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:34.442337 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:34.442388 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:34.442474 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:45:34.442747 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:34.443262 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:34.444310 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:34.444812 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:34.444933 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:34.444995 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:34.445255 (+ 260us) server_negotiation.cc:299] Negotiation successful
0812 09:45:34.445341 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260812 09:45:34.445627 21205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1993087690b402bbfb7d59c4acf941f" candidate_uuid: "ab93569b0cbf420da1da9a02c5c9303d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc730765a72740d0aec85da439bb5545" is_pre_election: true
I20260812 09:45:34.445950 21205 raft_consensus.cc:2454] T e1993087690b402bbfb7d59c4acf941f P fc730765a72740d0aec85da439bb5545 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab93569b0cbf420da1da9a02c5c9303d in term 0.
I20260812 09:45:34.446197 20790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1993087690b402bbfb7d59c4acf941f" candidate_uuid: "ab93569b0cbf420da1da9a02c5c9303d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d694b8f9ab224e7ea52d5303781494c5" is_pre_election: true
I20260812 09:45:34.446470 20881 leader_election.cc:302] T e1993087690b402bbfb7d59c4acf941f P ab93569b0cbf420da1da9a02c5c9303d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab93569b0cbf420da1da9a02c5c9303d, fc730765a72740d0aec85da439bb5545; no voters:
I20260812 09:45:34.446506 20790 raft_consensus.cc:2454] T e1993087690b402bbfb7d59c4acf941f P d694b8f9ab224e7ea52d5303781494c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab93569b0cbf420da1da9a02c5c9303d in term 0.
I20260812 09:45:34.446802 21326 raft_consensus.cc:2781] T e1993087690b402bbfb7d59c4acf941f P ab93569b0cbf420da1da9a02c5c9303d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 09:45:34.446908 21326 raft_consensus.cc:482] T e1993087690b402bbfb7d59c4acf941f P ab93569b0cbf420da1da9a02c5c9303d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 09:45:34.446960 21326 raft_consensus.cc:3037] T e1993087690b402bbfb7d59c4acf941f P ab93569b0cbf420da1da9a02c5c9303d [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:45:34.448019 21326 raft_consensus.cc:504] T e1993087690b402bbfb7d59c4acf941f P ab93569b0cbf420da1da9a02c5c9303d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc730765a72740d0aec85da439bb5545" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 44285 } } peers { permanent_uuid: "ab93569b0cbf420da1da9a02c5c9303d" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 41419 } } peers { permanent_uuid: "d694b8f9ab224e7ea52d5303781494c5" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 37255 } }
I20260812 09:45:34.448402 21326 leader_election.cc:288] T e1993087690b402bbfb7d59c4acf941f P ab93569b0cbf420da1da9a02c5c9303d [CANDIDATE]: Term 1 election: Requested vote from peers fc730765a72740d0aec85da439bb5545 (127.13.99.3:44285), d694b8f9ab224e7ea52d5303781494c5 (127.13.99.1:37255)
I20260812 09:45:34.448760 21205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1993087690b402bbfb7d59c4acf941f" candidate_uuid: "ab93569b0cbf420da1da9a02c5c9303d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc730765a72740d0aec85da439bb5545"
I20260812 09:45:34.448835 20790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1993087690b402bbfb7d59c4acf941f" candidate_uuid: "ab93569b0cbf420da1da9a02c5c9303d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d694b8f9ab224e7ea52d5303781494c5"
I20260812 09:45:34.448911 21205 raft_consensus.cc:3037] T e1993087690b402bbfb7d59c4acf941f P fc730765a72740d0aec85da439bb5545 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:45:34.448968 20790 raft_consensus.cc:3037] T e1993087690b402bbfb7d59c4acf941f P d694b8f9ab224e7ea52d5303781494c5 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:45:34.449926 21205 raft_consensus.cc:2454] T e1993087690b402bbfb7d59c4acf941f P fc730765a72740d0aec85da439bb5545 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab93569b0cbf420da1da9a02c5c9303d in term 1.
I20260812 09:45:34.450083 20790 raft_consensus.cc:2454] T e1993087690b402bbfb7d59c4acf941f P d694b8f9ab224e7ea52d5303781494c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab93569b0cbf420da1da9a02c5c9303d in term 1.
I20260812 09:45:34.450240 20881 leader_election.cc:302] T e1993087690b402bbfb7d59c4acf941f P ab93569b0cbf420da1da9a02c5c9303d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab93569b0cbf420da1da9a02c5c9303d, fc730765a72740d0aec85da439bb5545; no voters:
I20260812 09:45:34.450423 21326 raft_consensus.cc:2781] T e1993087690b402bbfb7d59c4acf941f P ab93569b0cbf420da1da9a02c5c9303d [term 1 FOLLOWER]: Leader election won for term 1
I20260812 09:45:34.450678 21326 raft_consensus.cc:686] T e1993087690b402bbfb7d59c4acf941f P ab93569b0cbf420da1da9a02c5c9303d [term 1 LEADER]: Becoming Leader. State: Replica: ab93569b0cbf420da1da9a02c5c9303d, State: Running, Role: LEADER
I20260812 09:45:34.451011 21326 consensus_queue.cc:237] T e1993087690b402bbfb7d59c4acf941f P ab93569b0cbf420da1da9a02c5c9303d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc730765a72740d0aec85da439bb5545" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 44285 } } peers { permanent_uuid: "ab93569b0cbf420da1da9a02c5c9303d" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 41419 } } peers { permanent_uuid: "d694b8f9ab224e7ea52d5303781494c5" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 37255 } }
I20260812 09:45:34.454407 20555 catalog_manager.cc:5495] T e1993087690b402bbfb7d59c4acf941f P ab93569b0cbf420da1da9a02c5c9303d reported cstate change: term changed from 0 to 1, leader changed from <none> to ab93569b0cbf420da1da9a02c5c9303d (127.13.99.2). New cstate: current_term: 1 leader_uuid: "ab93569b0cbf420da1da9a02c5c9303d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc730765a72740d0aec85da439bb5545" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 44285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab93569b0cbf420da1da9a02c5c9303d" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 41419 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d694b8f9ab224e7ea52d5303781494c5" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 37255 } health_report { overall_health: UNKNOWN } } }
I20260812 09:45:34.464985 21311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:34.461681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45100 (local address 127.13.99.2:41419)
0812 09:45:34.461800 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:34.461804 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:34.461827 (+ 23us) server_negotiation.cc:407] Connection header received
0812 09:45:34.461995 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:34.461998 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:34.462049 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:34.462135 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 09:45:34.462355 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:34.462828 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:34.463633 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:34.463839 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:34.463935 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:34.464198 (+ 263us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 09:45:34.464667 (+ 469us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 09:45:34.464757 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:34.464826 (+ 69us) server_negotiation.cc:299] Negotiation successful
0812 09:45:34.464871 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260812 09:45:34.470546 21205 raft_consensus.cc:1261] T e1993087690b402bbfb7d59c4acf941f P fc730765a72740d0aec85da439bb5545 [term 1 FOLLOWER]: Refusing update from remote peer ab93569b0cbf420da1da9a02c5c9303d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 09:45:34.470546 20790 raft_consensus.cc:1261] T e1993087690b402bbfb7d59c4acf941f P d694b8f9ab224e7ea52d5303781494c5 [term 1 FOLLOWER]: Refusing update from remote peer ab93569b0cbf420da1da9a02c5c9303d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 09:45:34.471263 21326 consensus_queue.cc:1035] T e1993087690b402bbfb7d59c4acf941f P ab93569b0cbf420da1da9a02c5c9303d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc730765a72740d0aec85da439bb5545" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 44285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 09:45:34.471436 21330 consensus_queue.cc:1035] T e1993087690b402bbfb7d59c4acf941f P ab93569b0cbf420da1da9a02c5c9303d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d694b8f9ab224e7ea52d5303781494c5" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 37255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 09:45:34.480885 21337 mvcc.cc:205] Tried to move back new op lower bound from 7317618419584327680 to 7317618419513802752. Current Snapshot: MvccSnapshot[applied={T|T < 7317618419584327680}]
I20260812 09:45:34.488883 20557 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45082:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 09:45:34.489053 20557 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45082:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 09:45:34.491480 20557 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 761849dbb1644dd9a894f70db486b1b1: Sending DeleteTablet for 3 replicas of tablet e1993087690b402bbfb7d59c4acf941f
I20260812 09:45:34.492329 20953 tablet_service.cc:1514] Processing DeleteTablet for tablet e1993087690b402bbfb7d59c4acf941f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 09:45:34 UTC) from {username='kudu', principal='kudu/127.13.99.62@KRBTEST.COM'} at 127.0.0.1:45096
I20260812 09:45:34.492367 20756 tablet_service.cc:1514] Processing DeleteTablet for tablet e1993087690b402bbfb7d59c4acf941f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 09:45:34 UTC) from {username='kudu', principal='kudu/127.13.99.62@KRBTEST.COM'} at 127.0.0.1:54262
I20260812 09:45:34.492348 21174 tablet_service.cc:1514] Processing DeleteTablet for tablet e1993087690b402bbfb7d59c4acf941f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 09:45:34 UTC) from {username='kudu', principal='kudu/127.13.99.62@KRBTEST.COM'} at 127.0.0.1:34826
I20260812 09:45:34.492868 21353 tablet_replica.cc:330] T e1993087690b402bbfb7d59c4acf941f P fc730765a72740d0aec85da439bb5545: stopping tablet replica
I20260812 09:45:34.493163 21350 tablet_replica.cc:330] T e1993087690b402bbfb7d59c4acf941f P ab93569b0cbf420da1da9a02c5c9303d: stopping tablet replica
I20260812 09:45:34.493222 21353 raft_consensus.cc:2229] T e1993087690b402bbfb7d59c4acf941f P fc730765a72740d0aec85da439bb5545 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 09:45:34.493427 21350 raft_consensus.cc:2229] T e1993087690b402bbfb7d59c4acf941f P ab93569b0cbf420da1da9a02c5c9303d [term 1 LEADER]: Raft consensus shutting down.
I20260812 09:45:34.493526 21353 raft_consensus.cc:2258] T e1993087690b402bbfb7d59c4acf941f P fc730765a72740d0aec85da439bb5545 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 09:45:34.493571 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 20662
I20260812 09:45:34.504699 21350 raft_consensus.cc:2258] T e1993087690b402bbfb7d59c4acf941f P ab93569b0cbf420da1da9a02c5c9303d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 09:45:34.504947 21353 ts_tablet_manager.cc:1905] T e1993087690b402bbfb7d59c4acf941f P fc730765a72740d0aec85da439bb5545: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 09:45:34.506083 21350 ts_tablet_manager.cc:1905] T e1993087690b402bbfb7d59c4acf941f P ab93569b0cbf420da1da9a02c5c9303d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 09:45:34.507766 21353 ts_tablet_manager.cc:1918] T e1993087690b402bbfb7d59c4acf941f P fc730765a72740d0aec85da439bb5545: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 09:45:34.507871 21353 log.cc:1196] T e1993087690b402bbfb7d59c4acf941f P fc730765a72740d0aec85da439bb5545: Deleting WAL directory at /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/minicluster-data/ts-2/wal/wals/e1993087690b402bbfb7d59c4acf941f
I20260812 09:45:34.508214 21353 ts_tablet_manager.cc:1939] T e1993087690b402bbfb7d59c4acf941f P fc730765a72740d0aec85da439bb5545: Deleting consensus metadata
I20260812 09:45:34.509595 20530 catalog_manager.cc:4841] TS fc730765a72740d0aec85da439bb5545 (127.13.99.3:44285): tablet e1993087690b402bbfb7d59c4acf941f (table test-table [id=af741b660e3d4b2d92639a51c6d89b3b]) successfully deleted
I20260812 09:45:34.510275 21350 ts_tablet_manager.cc:1918] T e1993087690b402bbfb7d59c4acf941f P ab93569b0cbf420da1da9a02c5c9303d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 09:45:34.510433 21350 log.cc:1196] T e1993087690b402bbfb7d59c4acf941f P ab93569b0cbf420da1da9a02c5c9303d: Deleting WAL directory at /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786527905902985-13708-0/minicluster-data/ts-1/wal/wals/e1993087690b402bbfb7d59c4acf941f
I20260812 09:45:34.510856 21350 ts_tablet_manager.cc:1939] T e1993087690b402bbfb7d59c4acf941f P ab93569b0cbf420da1da9a02c5c9303d: Deleting consensus metadata
W20260812 09:45:34.511915 20535 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.99.1:37255 (error 108)
I20260812 09:45:34.512079 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 20860
I20260812 09:45:34.517642 21302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:34.517132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.1:37255 (local address 127.0.0.1:54276)
0812 09:45:34.517407 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:34.517513 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.1:37255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154}
W20260812 09:45:34.518046 20535 catalog_manager.cc:4584] TS d694b8f9ab224e7ea52d5303781494c5 (127.13.99.1:37255): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e1993087690b402bbfb7d59c4acf941f: Network error: Client connection negotiation failed: client connection to 127.13.99.1:37255: connect: Connection refused (error 111)
I20260812 09:45:34.526917 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 21067
I20260812 09:45:34.535912 21302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:34.527126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.2:41419 (local address 127.0.0.1:45108)
0812 09:45:34.535742 (+ 8616us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:34.535813 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.2:41419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":8448}
I20260812 09:45:34.541879 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 20507
2026-08-12T09:45:34Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2573 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-12T09:45:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T09:45:34Z Disabled control of system clock
I20260812 09:45:34.584095 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786527905902985-13708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786527905902985-13708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786527905902985-13708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.62:41647
--webserver_interface=127.13.99.62
--webserver_port=0
--builtin_ntp_servers=127.13.99.20:43189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.99.62:41647
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260812 09:45:34.699295 21359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:45:34.699669 21359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:45:34.699713 21359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:45:34.703821 21359 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 09:45:34.703877 21359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:45:34.703898 21359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:45:34.703917 21359 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 09:45:34.703930 21359 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 09:45:34.708631 21359 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:43189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786527905902985-13708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786527905902985-13708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.99.62:41647
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.62:41647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.99.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21359
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786527905902985-13708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:45:34.709741 21359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:45:34.711117 21359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:45:34.719671 21366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:34.719789 21368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:34.719681 21370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:45:34.720402 21359 server_base.cc:1034] running on GCE node
I20260812 09:45:34.721213 21359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:45:34.722647 21359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:45:34.723937 21359 hybrid_clock.cc:648] HybridClock initialized: now 1786527934723920 us; error 62 us; skew 500 ppm
I20260812 09:45:34.725916 21359 webserver.cc:460] Webserver started at http://127.13.99.62:35677/ using document root <none> and password file <none>
I20260812 09:45:34.726518 21359 fs_manager.cc:359] Metadata directory not provided
I20260812 09:45:34.726648 21359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:45:34.726902 21359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:45:34.729089 21359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786527905902985-13708-0/minicluster-data/master-0/data/instance:
uuid: "e58f66a1a95947fb844738fbf18f09ab"
format_stamp: "Formatted at 2026-08-12 09:45:34 on dist-test-slave-jq4t"
server_key: "71814fc74a01e3f664d67635f2cf1566"
server_key_iv: "b7ca70d5ff188342d3dfc5a969dbbd95"
server_key_version: "encryptionkey@0"
I20260812 09:45:34.729666 21359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786527905902985-13708-0/minicluster-data/master-0/wal/instance:
uuid: "e58f66a1a95947fb844738fbf18f09ab"
format_stamp: "Formatted at 2026-08-12 09:45:34 on dist-test-slave-jq4t"
server_key: "71814fc74a01e3f664d67635f2cf1566"
server_key_iv: "b7ca70d5ff188342d3dfc5a969dbbd95"
server_key_version: "encryptionkey@0"
I20260812 09:45:34.733385 21359 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 09:45:34.735740 21381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:34.737026 21359 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 09:45:34.737177 21359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786527905902985-13708-0/minicluster-data/master-0/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786527905902985-13708-0/minicluster-data/master-0/wal
uuid: "e58f66a1a95947fb844738fbf18f09ab"
format_stamp: "Formatted at 2026-08-12 09:45:34 on dist-test-slave-jq4t"
server_key: "71814fc74a01e3f664d67635f2cf1566"
server_key_iv: "b7ca70d5ff188342d3dfc5a969dbbd95"
server_key_version: "encryptionkey@0"
I20260812 09:45:34.737291 21359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786527905902985-13708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786527905902985-13708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786527905902985-13708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:45:34.741936 21359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:45:34.742537 21359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:45:34.742717 21359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:45:34.750092 21359 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.62:41647
I20260812 09:45:34.750099 21462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.62:41647 every 8 connection(s)
I20260812 09:45:34.751104 21359 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
I20260812 09:45:34.754585 21463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:45:34.760097 21463 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e58f66a1a95947fb844738fbf18f09ab: Bootstrap starting.
I20260812 09:45:34.760684 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 21359
I20260812 09:45:34.760826 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786527905902985-13708-0/minicluster-data/master-0/wal/instance
I20260812 09:45:34.761209 13708 external_mini_cluster.cc:1428] Setting key 5bab65ed602bc9dc4efc5c1fd8e53f4c
I20260812 09:45:34.763213 21463 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e58f66a1a95947fb844738fbf18f09ab: Neither blocks nor log segments found. Creating new log.
I20260812 09:45:34.764020 21463 log.cc:824] T 00000000000000000000000000000000 P e58f66a1a95947fb844738fbf18f09ab: Log is configured to *not* fsync() on all Append() calls
I20260812 09:45:34.766476 21463 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e58f66a1a95947fb844738fbf18f09ab: No bootstrap required, opened a new log
I20260812 09:45:34.767225 21467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:34.762628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34986 (local address 127.13.99.62:41647)
0812 09:45:34.763315 (+ 687us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:34.763359 (+ 44us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:34.763410 (+ 51us) server_negotiation.cc:407] Connection header received
0812 09:45:34.764075 (+ 665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:34.764093 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:34.764595 (+ 502us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:34.764970 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:34.765835 (+ 865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:34.765839 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:34.765852 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 09:45:34.765862 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:34.765932 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 09:45:34.765949 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:34.766040 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:34.766186 (+ 146us) server_negotiation.cc:299] Negotiation successful
0812 09:45:34.766449 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":143,"threads_started":1}
I20260812 09:45:34.770546 21463 raft_consensus.cc:348] T 00000000000000000000000000000000 P e58f66a1a95947fb844738fbf18f09ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e58f66a1a95947fb844738fbf18f09ab" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 41647 } }
I20260812 09:45:34.770790 21463 raft_consensus.cc:374] T 00000000000000000000000000000000 P e58f66a1a95947fb844738fbf18f09ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:45:34.770898 21463 raft_consensus.cc:729] T 00000000000000000000000000000000 P e58f66a1a95947fb844738fbf18f09ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e58f66a1a95947fb844738fbf18f09ab, State: Initialized, Role: FOLLOWER
I20260812 09:45:34.771323 21463 consensus_queue.cc:260] T 00000000000000000000000000000000 P e58f66a1a95947fb844738fbf18f09ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e58f66a1a95947fb844738fbf18f09ab" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 41647 } }
I20260812 09:45:34.771481 21463 raft_consensus.cc:388] T 00000000000000000000000000000000 P e58f66a1a95947fb844738fbf18f09ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 09:45:34.771582 21463 raft_consensus.cc:482] T 00000000000000000000000000000000 P e58f66a1a95947fb844738fbf18f09ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 09:45:34.771703 21463 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e58f66a1a95947fb844738fbf18f09ab [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:45:34.772879 21463 raft_consensus.cc:504] T 00000000000000000000000000000000 P e58f66a1a95947fb844738fbf18f09ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e58f66a1a95947fb844738fbf18f09ab" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 41647 } }
I20260812 09:45:34.773231 21463 leader_election.cc:302] T 00000000000000000000000000000000 P e58f66a1a95947fb844738fbf18f09ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e58f66a1a95947fb844738fbf18f09ab; no voters:
I20260812 09:45:34.773542 21463 leader_election.cc:288] T 00000000000000000000000000000000 P e58f66a1a95947fb844738fbf18f09ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 09:45:34.773641 21470 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e58f66a1a95947fb844738fbf18f09ab [term 1 FOLLOWER]: Leader election won for term 1
I20260812 09:45:34.773936 21470 raft_consensus.cc:686] T 00000000000000000000000000000000 P e58f66a1a95947fb844738fbf18f09ab [term 1 LEADER]: Becoming Leader. State: Replica: e58f66a1a95947fb844738fbf18f09ab, State: Running, Role: LEADER
I20260812 09:45:34.774247 21470 consensus_queue.cc:237] T 00000000000000000000000000000000 P e58f66a1a95947fb844738fbf18f09ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e58f66a1a95947fb844738fbf18f09ab" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 41647 } }
I20260812 09:45:34.774981 21463 sys_catalog.cc:564] T 00000000000000000000000000000000 P e58f66a1a95947fb844738fbf18f09ab [sys.catalog]: configured and running, proceeding with master startup.
I20260812 09:45:34.776307 21474 sys_catalog.cc:455] T 00000000000000000000000000000000 P e58f66a1a95947fb844738fbf18f09ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader e58f66a1a95947fb844738fbf18f09ab. Latest consensus state: current_term: 1 leader_uuid: "e58f66a1a95947fb844738fbf18f09ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e58f66a1a95947fb844738fbf18f09ab" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 41647 } } }
I20260812 09:45:34.776459 21474 sys_catalog.cc:458] T 00000000000000000000000000000000 P e58f66a1a95947fb844738fbf18f09ab [sys.catalog]: This master's current role is: LEADER
I20260812 09:45:34.776794 21472 sys_catalog.cc:455] T 00000000000000000000000000000000 P e58f66a1a95947fb844738fbf18f09ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e58f66a1a95947fb844738fbf18f09ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e58f66a1a95947fb844738fbf18f09ab" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 41647 } } }
I20260812 09:45:34.776892 21472 sys_catalog.cc:458] T 00000000000000000000000000000000 P e58f66a1a95947fb844738fbf18f09ab [sys.catalog]: This master's current role is: LEADER
I20260812 09:45:34.777024 21491 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 09:45:34.780238 21491 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 09:45:34.786005 21491 catalog_manager.cc:1347] Generated new cluster ID: e2be5bac5f63468fad1345b94ac8cd29
I20260812 09:45:34.786082 21491 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 09:45:34.809332 21491 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 09:45:34.809659 21491 catalog_manager.cc:1504] Loading token signing keys...
I20260812 09:45:34.817739 21491 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e58f66a1a95947fb844738fbf18f09ab: Generated new TSK 0
I20260812 09:45:34.818397 21491 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 09:45:34.824839 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786527905902985-13708-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786527905902985-13708-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786527905902985-13708-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786527905902985-13708-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.99.1:0
--local_ip_for_outbound_sockets=127.13.99.1
--webserver_interface=127.13.99.1
--webserver_port=0
--tserver_master_addrs=127.13.99.62:41647
--builtin_ntp_servers=127.13.99.20:43189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260812 09:45:34.961395 21509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:45:34.961644 21509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:45:34.961683 21509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:45:34.965653 21509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:45:34.965701 21509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:45:34.965755 21509 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.99.1
I20260812 09:45:34.970275 21509 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:43189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786527905902985-13708-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786527905902985-13708-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.99.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786527905902985-13708-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.99.1
--webserver_port=0
--tserver_master_addrs=127.13.99.62:41647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21509
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.99.1
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786527905902985-13708-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:45:34.971279 21509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:45:34.972704 21509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:45:34.982511 21517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:45:34.982600 21509 server_base.cc:1034] running on GCE node
W20260812 09:45:34.982518 21521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:34.982517 21519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:45:34.983098 21509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:45:34.983728 21509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:45:34.984916 21509 hybrid_clock.cc:648] HybridClock initialized: now 1786527934984898 us; error 29 us; skew 500 ppm
I20260812 09:45:34.986797 21509 webserver.cc:460] Webserver started at http://127.13.99.1:37089/ using document root <none> and password file <none>
I20260812 09:45:34.987408 21509 fs_manager.cc:359] Metadata directory not provided
I20260812 09:45:34.987504 21509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:45:34.987748 21509 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:45:34.989933 21509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786527905902985-13708-0/minicluster-data/ts-0/data/instance:
uuid: "0f50df1481944fa5840dd2c440a5e747"
format_stamp: "Formatted at 2026-08-12 09:45:34 on dist-test-slave-jq4t"
server_key: "389d42d2c87c1d7e2962d5274113a6f3"
server_key_iv: "8a2422c8e77a6150a1803b37802e387a"
server_key_version: "encryptionkey@0"
I20260812 09:45:34.990511 21509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786527905902985-13708-0/minicluster-data/ts-0/wal/instance:
uuid: "0f50df1481944fa5840dd2c440a5e747"
format_stamp: "Formatted at 2026-08-12 09:45:34 on dist-test-slave-jq4t"
server_key: "389d42d2c87c1d7e2962d5274113a6f3"
server_key_iv: "8a2422c8e77a6150a1803b37802e387a"
server_key_version: "encryptionkey@0"
I20260812 09:45:34.994374 21509 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 09:45:34.996979 21527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:34.998351 21509 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 09:45:34.998507 21509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786527905902985-13708-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786527905902985-13708-0/minicluster-data/ts-0/wal
uuid: "0f50df1481944fa5840dd2c440a5e747"
format_stamp: "Formatted at 2026-08-12 09:45:34 on dist-test-slave-jq4t"
server_key: "389d42d2c87c1d7e2962d5274113a6f3"
server_key_iv: "8a2422c8e77a6150a1803b37802e387a"
server_key_version: "encryptionkey@0"
I20260812 09:45:34.998632 21509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786527905902985-13708-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786527905902985-13708-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786527905902985-13708-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:45:35.003327 21509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:45:35.003937 21509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:45:35.004122 21509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:45:35.004758 21509 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 09:45:35.005679 21509 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:45:35.005751 21509 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:35.005823 21509 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:45:35.005867 21509 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:35.014629 21509 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.1:44355
I20260812 09:45:35.014665 21696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.1:44355 every 8 connection(s)
I20260812 09:45:35.015563 21509 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786527905902985-13708-0/minicluster-data/ts-0/data/info.pb
I20260812 09:45:35.021564 21467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:35.017442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.1:40783 (local address 127.13.99.62:41647)
0812 09:45:35.017588 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:35.017592 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:35.018595 (+ 1003us) server_negotiation.cc:407] Connection header received
0812 09:45:35.019902 (+ 1307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:35.019905 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:35.019946 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:35.020028 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:35.021161 (+ 1133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:35.021164 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:35.021166 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 09:45:35.021179 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:35.021199 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 09:45:35.021207 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:35.021290 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:35.021416 (+ 126us) server_negotiation.cc:299] Negotiation successful
0812 09:45:35.021457 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260812 09:45:35.022415 21706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:35.017779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:41647 (local address 127.13.99.1:40783)
0812 09:45:35.018443 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:35.018484 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:35.019702 (+ 1218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:35.020200 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:35.020207 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:35.020292 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:45:35.020873 (+ 581us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 09:45:35.020874 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:45:35.020914 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 09:45:35.020999 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:45:35.021288 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:45:35.021296 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:45:35.021300 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 09:45:35.021428 (+ 128us) client_negotiation.cc:240] Negotiation successful
0812 09:45:35.021645 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":122,"threads_started":1}
I20260812 09:45:35.023006 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 21509
I20260812 09:45:35.023151 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786527905902985-13708-0/minicluster-data/ts-0/wal/instance
I20260812 09:45:35.023445 13708 external_mini_cluster.cc:1428] Setting key 12b768f8e25637540348ff0d6b398cd9
I20260812 09:45:35.023892 21700 heartbeater.cc:347] Connected to a master server at 127.13.99.62:41647
I20260812 09:45:35.024140 21700 heartbeater.cc:464] Registering TS with master...
I20260812 09:45:35.024657 21700 heartbeater.cc:511] Master 127.13.99.62:41647 requested a full tablet report, sending...
I20260812 09:45:35.025475 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786527905902985-13708-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786527905902985-13708-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786527905902985-13708-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786527905902985-13708-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.99.2:0
--local_ip_for_outbound_sockets=127.13.99.2
--webserver_interface=127.13.99.2
--webserver_port=0
--tserver_master_addrs=127.13.99.62:41647
--builtin_ntp_servers=127.13.99.20:43189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260812 09:45:35.026355 21409 ts_manager.cc:194] Registered new tserver with Master: 0f50df1481944fa5840dd2c440a5e747 (127.13.99.1:44355)
W20260812 09:45:35.141256 21707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:45:35.141525 21707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:45:35.141570 21707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:45:35.145608 21707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:45:35.145665 21707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:45:35.145720 21707 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.99.2
I20260812 09:45:35.150523 21707 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:43189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786527905902985-13708-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786527905902985-13708-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.99.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786527905902985-13708-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.99.2
--webserver_port=0
--tserver_master_addrs=127.13.99.62:41647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21707
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.99.2
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786527905902985-13708-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:45:35.151686 21707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:45:35.153196 21707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:45:35.162937 21713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:45:35.162992 21707 server_base.cc:1034] running on GCE node
W20260812 09:45:35.162935 21717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:35.162931 21715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:45:35.163578 21707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:45:35.164218 21707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:45:35.165405 21707 hybrid_clock.cc:648] HybridClock initialized: now 1786527935165386 us; error 72 us; skew 500 ppm
I20260812 09:45:35.167279 21707 webserver.cc:460] Webserver started at http://127.13.99.2:42813/ using document root <none> and password file <none>
I20260812 09:45:35.167872 21707 fs_manager.cc:359] Metadata directory not provided
I20260812 09:45:35.167968 21707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:45:35.168215 21707 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:45:35.170313 21707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786527905902985-13708-0/minicluster-data/ts-1/data/instance:
uuid: "b47206cd1be5454f97b1efada57a693e"
format_stamp: "Formatted at 2026-08-12 09:45:35 on dist-test-slave-jq4t"
server_key: "ef01b4945cc205095938cc44373b79d8"
server_key_iv: "850a0bfbc6a014d86b765d9185791c91"
server_key_version: "encryptionkey@0"
I20260812 09:45:35.170905 21707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786527905902985-13708-0/minicluster-data/ts-1/wal/instance:
uuid: "b47206cd1be5454f97b1efada57a693e"
format_stamp: "Formatted at 2026-08-12 09:45:35 on dist-test-slave-jq4t"
server_key: "ef01b4945cc205095938cc44373b79d8"
server_key_iv: "850a0bfbc6a014d86b765d9185791c91"
server_key_version: "encryptionkey@0"
I20260812 09:45:35.174674 21707 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260812 09:45:35.177570 21727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:35.178793 21707 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 09:45:35.178925 21707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786527905902985-13708-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786527905902985-13708-0/minicluster-data/ts-1/wal
uuid: "b47206cd1be5454f97b1efada57a693e"
format_stamp: "Formatted at 2026-08-12 09:45:35 on dist-test-slave-jq4t"
server_key: "ef01b4945cc205095938cc44373b79d8"
server_key_iv: "850a0bfbc6a014d86b765d9185791c91"
server_key_version: "encryptionkey@0"
I20260812 09:45:35.179036 21707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786527905902985-13708-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786527905902985-13708-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786527905902985-13708-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:45:35.183673 21707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:45:35.184262 21707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:45:35.184448 21707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:45:35.185000 21707 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 09:45:35.185938 21707 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:45:35.185985 21707 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:35.186055 21707 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:45:35.186094 21707 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:35.194787 21707 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.2:43629
I20260812 09:45:35.194813 21893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.2:43629 every 8 connection(s)
I20260812 09:45:35.195665 21707 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786527905902985-13708-0/minicluster-data/ts-1/data/info.pb
I20260812 09:45:35.201855 21467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:35.197554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.2:60209 (local address 127.13.99.62:41647)
0812 09:45:35.197725 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:35.197730 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:35.198772 (+ 1042us) server_negotiation.cc:407] Connection header received
0812 09:45:35.200075 (+ 1303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:35.200087 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:35.200129 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:35.200215 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:35.201345 (+ 1130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:35.201348 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:35.201350 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 09:45:35.201363 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:35.201384 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 09:45:35.201393 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:35.201481 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:35.201657 (+ 176us) server_negotiation.cc:299] Negotiation successful
0812 09:45:35.201711 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260812 09:45:35.202837 21898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:35.197978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:41647 (local address 127.13.99.2:60209)
0812 09:45:35.198631 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:35.198669 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:35.199836 (+ 1167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:35.200383 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:35.200389 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:35.200529 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:45:35.201091 (+ 562us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 09:45:35.201093 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:45:35.201132 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 09:45:35.201199 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:45:35.201478 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:45:35.201485 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:45:35.201489 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 09:45:35.201696 (+ 207us) client_negotiation.cc:240] Negotiation successful
0812 09:45:35.201934 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":115,"threads_started":1}
I20260812 09:45:35.203086 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 21707
I20260812 09:45:35.203229 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786527905902985-13708-0/minicluster-data/ts-1/wal/instance
I20260812 09:45:35.203568 13708 external_mini_cluster.cc:1428] Setting key c52b9ebe76e82f237312e66e1d1153f2
I20260812 09:45:35.204056 21895 heartbeater.cc:347] Connected to a master server at 127.13.99.62:41647
I20260812 09:45:35.204303 21895 heartbeater.cc:464] Registering TS with master...
I20260812 09:45:35.204829 21895 heartbeater.cc:511] Master 127.13.99.62:41647 requested a full tablet report, sending...
I20260812 09:45:35.205917 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786527905902985-13708-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786527905902985-13708-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786527905902985-13708-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786527905902985-13708-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.99.3:0
--local_ip_for_outbound_sockets=127.13.99.3
--webserver_interface=127.13.99.3
--webserver_port=0
--tserver_master_addrs=127.13.99.62:41647
--builtin_ntp_servers=127.13.99.20:43189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260812 09:45:35.206125 21409 ts_manager.cc:194] Registered new tserver with Master: b47206cd1be5454f97b1efada57a693e (127.13.99.2:43629)
W20260812 09:45:35.321456 21900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:45:35.321697 21900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:45:35.321734 21900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:45:35.325704 21900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:45:35.325752 21900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:45:35.325803 21900 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.99.3
I20260812 09:45:35.330391 21900 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:43189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786527905902985-13708-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786527905902985-13708-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.99.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786527905902985-13708-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.99.3
--webserver_port=0
--tserver_master_addrs=127.13.99.62:41647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21900
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.99.3
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786527905902985-13708-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:45:35.331367 21900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:45:35.332752 21900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:45:35.342657 21909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:35.342684 21907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:35.342700 21905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:45:35.343091 21900 server_base.cc:1034] running on GCE node
I20260812 09:45:35.343441 21900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:45:35.344066 21900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:45:35.345253 21900 hybrid_clock.cc:648] HybridClock initialized: now 1786527935345234 us; error 30 us; skew 500 ppm
I20260812 09:45:35.347090 21900 webserver.cc:460] Webserver started at http://127.13.99.3:46357/ using document root <none> and password file <none>
I20260812 09:45:35.347647 21900 fs_manager.cc:359] Metadata directory not provided
I20260812 09:45:35.347708 21900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:45:35.347882 21900 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:45:35.349987 21900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786527905902985-13708-0/minicluster-data/ts-2/data/instance:
uuid: "78e4974170c34cf898aa560c9da7b69e"
format_stamp: "Formatted at 2026-08-12 09:45:35 on dist-test-slave-jq4t"
server_key: "83289a92f05579f23691cb3e13bf30ea"
server_key_iv: "f855826435a07e0466a30db89079ef4a"
server_key_version: "encryptionkey@0"
I20260812 09:45:35.350517 21900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786527905902985-13708-0/minicluster-data/ts-2/wal/instance:
uuid: "78e4974170c34cf898aa560c9da7b69e"
format_stamp: "Formatted at 2026-08-12 09:45:35 on dist-test-slave-jq4t"
server_key: "83289a92f05579f23691cb3e13bf30ea"
server_key_iv: "f855826435a07e0466a30db89079ef4a"
server_key_version: "encryptionkey@0"
I20260812 09:45:35.354439 21900 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260812 09:45:35.357213 21917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:35.358584 21900 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 09:45:35.358744 21900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786527905902985-13708-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786527905902985-13708-0/minicluster-data/ts-2/wal
uuid: "78e4974170c34cf898aa560c9da7b69e"
format_stamp: "Formatted at 2026-08-12 09:45:35 on dist-test-slave-jq4t"
server_key: "83289a92f05579f23691cb3e13bf30ea"
server_key_iv: "f855826435a07e0466a30db89079ef4a"
server_key_version: "encryptionkey@0"
I20260812 09:45:35.358902 21900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786527905902985-13708-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786527905902985-13708-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786527905902985-13708-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:45:35.363511 21900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:45:35.364157 21900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:45:35.364349 21900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:45:35.364945 21900 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 09:45:35.365862 21900 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:45:35.365944 21900 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:35.366019 21900 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:45:35.366062 21900 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:35.375435 21900 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.3:46023
I20260812 09:45:35.375463 22086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.3:46023 every 8 connection(s)
I20260812 09:45:35.376466 21900 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786527905902985-13708-0/minicluster-data/ts-2/data/info.pb
I20260812 09:45:35.383567 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 21900
I20260812 09:45:35.383721 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786527905902985-13708-0/minicluster-data/ts-2/wal/instance
I20260812 09:45:35.384042 13708 external_mini_cluster.cc:1428] Setting key a902b0b8da7f53d81cbbe11439951ac0
I20260812 09:45:35.386919 21467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:35.378406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.3:38999 (local address 127.13.99.62:41647)
0812 09:45:35.378572 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:35.378577 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:35.381466 (+ 2889us) server_negotiation.cc:407] Connection header received
0812 09:45:35.384594 (+ 3128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:35.384626 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:35.384686 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:35.384820 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:35.386359 (+ 1539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:35.386363 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:35.386366 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 09:45:35.386389 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:35.386414 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 09:45:35.386433 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:35.386533 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:35.386718 (+ 185us) server_negotiation.cc:299] Negotiation successful
0812 09:45:35.386771 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260812 09:45:35.388314 22091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:35.378857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:41647 (local address 127.13.99.3:38999)
0812 09:45:35.381241 (+ 2384us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:35.381306 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:35.383456 (+ 2150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:35.385103 (+ 1647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:35.385111 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:35.385217 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:45:35.386041 (+ 824us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 09:45:35.386044 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:45:35.386111 (+ 67us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 09:45:35.386214 (+ 103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:45:35.386535 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:45:35.386548 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:45:35.386558 (+ 10us) client_negotiation.cc:769] Sending connection context
0812 09:45:35.386772 (+ 214us) client_negotiation.cc:240] Negotiation successful
0812 09:45:35.387150 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1730,"thread_start_us":164,"threads_started":1}
I20260812 09:45:35.389631 22088 heartbeater.cc:347] Connected to a master server at 127.13.99.62:41647
I20260812 09:45:35.389910 22088 heartbeater.cc:464] Registering TS with master...
I20260812 09:45:35.390314 22088 heartbeater.cc:511] Master 127.13.99.62:41647 requested a full tablet report, sending...
I20260812 09:45:35.391312 21409 ts_manager.cc:194] Registered new tserver with Master: 78e4974170c34cf898aa560c9da7b69e (127.13.99.3:46023)
I20260812 09:45:35.399333 13708 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 09:45:35.403134 21467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:35.401332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59000 (local address 127.13.99.62:41647)
0812 09:45:35.401506 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:35.401511 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:35.401725 (+ 214us) server_negotiation.cc:407] Connection header received
0812 09:45:35.401821 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:35.401823 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:35.401860 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:35.401934 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:35.402755 (+ 821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:35.402759 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:35.402761 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 09:45:35.402772 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:35.402792 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 09:45:35.402800 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:35.402886 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:35.402934 (+ 48us) server_negotiation.cc:299] Negotiation successful
0812 09:45:35.402969 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260812 09:45:35.406076 21409 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59000:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 09:45:35.409119 21409 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 09:45:35.420851 22118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:35.417605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.3:46023 (local address 127.0.0.1:53828)
0812 09:45:35.418201 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:35.418218 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:35.418303 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:35.419493 (+ 1190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:35.419496 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:35.419513 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:45:35.419944 (+ 431us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 09:45:35.419948 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:45:35.419961 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 09:45:35.420004 (+ 43us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:45:35.420425 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:45:35.420435 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:45:35.420441 (+ 6us) client_negotiation.cc:769] Sending connection context
0812 09:45:35.420608 (+ 167us) client_negotiation.cc:240] Negotiation successful
0812 09:45:35.420690 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":73,"threads_started":1}
I20260812 09:45:35.420912 22122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:35.418528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53828 (local address 127.13.99.3:46023)
0812 09:45:35.419055 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:35.419068 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:35.419094 (+ 26us) server_negotiation.cc:407] Connection header received
0812 09:45:35.419151 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:35.419157 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:35.419345 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:35.419511 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:35.420251 (+ 740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:35.420256 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:35.420265 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 09:45:35.420273 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:35.420305 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 09:45:35.420322 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:35.420454 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:35.420652 (+ 198us) server_negotiation.cc:299] Negotiation successful
0812 09:45:35.420733 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":154,"threads_started":1}
I20260812 09:45:35.421597 22115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:35.417585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.1:44355 (local address 127.0.0.1:48856)
0812 09:45:35.417967 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:35.418014 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:35.418113 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:35.418903 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:35.418909 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:35.418988 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:45:35.419641 (+ 653us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 09:45:35.419643 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:45:35.419665 (+ 22us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 09:45:35.420006 (+ 341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:45:35.421329 (+ 1323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:45:35.421333 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:45:35.421336 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 09:45:35.421394 (+ 58us) client_negotiation.cc:240] Negotiation successful
0812 09:45:35.421431 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":120,"threads_started":1}
I20260812 09:45:35.422214 22121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:35.418345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.2:43629 (local address 127.0.0.1:44532)
0812 09:45:35.419132 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:35.419147 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:35.419230 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:35.420557 (+ 1327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:35.420562 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:35.420583 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:45:35.421290 (+ 707us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 09:45:35.421293 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:45:35.421306 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 09:45:35.421338 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:45:35.421982 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:45:35.421985 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:45:35.421989 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 09:45:35.422072 (+ 83us) client_negotiation.cc:240] Negotiation successful
0812 09:45:35.422117 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":711,"thread_start_us":116,"threads_started":1}
I20260812 09:45:35.422606 22126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:35.418702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44532 (local address 127.13.99.2:43629)
0812 09:45:35.419489 (+ 787us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:35.419501 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:35.419525 (+ 24us) server_negotiation.cc:407] Connection header received
0812 09:45:35.419591 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:35.419596 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:35.419762 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:35.419898 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:35.421510 (+ 1612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:35.421516 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:35.421528 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 09:45:35.421543 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:35.421588 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 09:45:35.421624 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:35.421927 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:35.422426 (+ 499us) server_negotiation.cc:299] Negotiation successful
0812 09:45:35.422488 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":667,"thread_start_us":150,"threads_started":1}
I20260812 09:45:35.424062 21987 tablet_service.cc:1467] Processing CreateTablet for tablet 78166c891e074df39b845a7d33299869 (DEFAULT_TABLE table=test-table [id=1b52d5379b154361be585d7e59d52d9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 09:45:35.424722 22120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:35.417727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48856 (local address 127.13.99.1:44355)
0812 09:45:35.418383 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:35.418403 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:35.418429 (+ 26us) server_negotiation.cc:407] Connection header received
0812 09:45:35.418504 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:35.418511 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:35.418709 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:35.420161 (+ 1452us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:35.421105 (+ 944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:35.421113 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:35.421127 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 09:45:35.421140 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:35.421183 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 09:45:35.421213 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:35.421885 (+ 672us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:35.421934 (+ 49us) server_negotiation.cc:299] Negotiation successful
0812 09:45:35.422006 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":182,"threads_started":1}
I20260812 09:45:35.425184 21987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78166c891e074df39b845a7d33299869. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:45:35.425175 21792 tablet_service.cc:1467] Processing CreateTablet for tablet 78166c891e074df39b845a7d33299869 (DEFAULT_TABLE table=test-table [id=1b52d5379b154361be585d7e59d52d9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 09:45:35.426360 21792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78166c891e074df39b845a7d33299869. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:45:35.427649 21594 tablet_service.cc:1467] Processing CreateTablet for tablet 78166c891e074df39b845a7d33299869 (DEFAULT_TABLE table=test-table [id=1b52d5379b154361be585d7e59d52d9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 09:45:35.428712 21594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78166c891e074df39b845a7d33299869. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:45:35.432632 22127 tablet_bootstrap.cc:492] T 78166c891e074df39b845a7d33299869 P b47206cd1be5454f97b1efada57a693e: Bootstrap starting.
I20260812 09:45:35.433985 22128 tablet_bootstrap.cc:492] T 78166c891e074df39b845a7d33299869 P 78e4974170c34cf898aa560c9da7b69e: Bootstrap starting.
I20260812 09:45:35.435439 22129 tablet_bootstrap.cc:492] T 78166c891e074df39b845a7d33299869 P 0f50df1481944fa5840dd2c440a5e747: Bootstrap starting.
I20260812 09:45:35.435498 22127 tablet_bootstrap.cc:654] T 78166c891e074df39b845a7d33299869 P b47206cd1be5454f97b1efada57a693e: Neither blocks nor log segments found. Creating new log.
I20260812 09:45:35.436290 22128 tablet_bootstrap.cc:654] T 78166c891e074df39b845a7d33299869 P 78e4974170c34cf898aa560c9da7b69e: Neither blocks nor log segments found. Creating new log.
I20260812 09:45:35.437001 22127 log.cc:824] T 78166c891e074df39b845a7d33299869 P b47206cd1be5454f97b1efada57a693e: Log is configured to *not* fsync() on all Append() calls
I20260812 09:45:35.437199 22128 log.cc:824] T 78166c891e074df39b845a7d33299869 P 78e4974170c34cf898aa560c9da7b69e: Log is configured to *not* fsync() on all Append() calls
I20260812 09:45:35.437332 22129 tablet_bootstrap.cc:654] T 78166c891e074df39b845a7d33299869 P 0f50df1481944fa5840dd2c440a5e747: Neither blocks nor log segments found. Creating new log.
I20260812 09:45:35.438185 22129 log.cc:824] T 78166c891e074df39b845a7d33299869 P 0f50df1481944fa5840dd2c440a5e747: Log is configured to *not* fsync() on all Append() calls
I20260812 09:45:35.439268 22127 tablet_bootstrap.cc:492] T 78166c891e074df39b845a7d33299869 P b47206cd1be5454f97b1efada57a693e: No bootstrap required, opened a new log
I20260812 09:45:35.439268 22128 tablet_bootstrap.cc:492] T 78166c891e074df39b845a7d33299869 P 78e4974170c34cf898aa560c9da7b69e: No bootstrap required, opened a new log
I20260812 09:45:35.439482 22128 ts_tablet_manager.cc:1397] T 78166c891e074df39b845a7d33299869 P 78e4974170c34cf898aa560c9da7b69e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260812 09:45:35.439481 22127 ts_tablet_manager.cc:1397] T 78166c891e074df39b845a7d33299869 P b47206cd1be5454f97b1efada57a693e: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260812 09:45:35.440198 22129 tablet_bootstrap.cc:492] T 78166c891e074df39b845a7d33299869 P 0f50df1481944fa5840dd2c440a5e747: No bootstrap required, opened a new log
I20260812 09:45:35.440387 22129 ts_tablet_manager.cc:1397] T 78166c891e074df39b845a7d33299869 P 0f50df1481944fa5840dd2c440a5e747: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260812 09:45:35.444041 22129 raft_consensus.cc:348] T 78166c891e074df39b845a7d33299869 P 0f50df1481944fa5840dd2c440a5e747 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f50df1481944fa5840dd2c440a5e747" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 44355 } } peers { permanent_uuid: "78e4974170c34cf898aa560c9da7b69e" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 46023 } } peers { permanent_uuid: "b47206cd1be5454f97b1efada57a693e" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 43629 } }
I20260812 09:45:35.444275 22129 raft_consensus.cc:374] T 78166c891e074df39b845a7d33299869 P 0f50df1481944fa5840dd2c440a5e747 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:45:35.444329 22129 raft_consensus.cc:729] T 78166c891e074df39b845a7d33299869 P 0f50df1481944fa5840dd2c440a5e747 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f50df1481944fa5840dd2c440a5e747, State: Initialized, Role: FOLLOWER
I20260812 09:45:35.444635 22128 raft_consensus.cc:348] T 78166c891e074df39b845a7d33299869 P 78e4974170c34cf898aa560c9da7b69e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f50df1481944fa5840dd2c440a5e747" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 44355 } } peers { permanent_uuid: "78e4974170c34cf898aa560c9da7b69e" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 46023 } } peers { permanent_uuid: "b47206cd1be5454f97b1efada57a693e" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 43629 } }
I20260812 09:45:35.444635 22127 raft_consensus.cc:348] T 78166c891e074df39b845a7d33299869 P b47206cd1be5454f97b1efada57a693e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f50df1481944fa5840dd2c440a5e747" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 44355 } } peers { permanent_uuid: "78e4974170c34cf898aa560c9da7b69e" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 46023 } } peers { permanent_uuid: "b47206cd1be5454f97b1efada57a693e" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 43629 } }
I20260812 09:45:35.444815 22129 consensus_queue.cc:260] T 78166c891e074df39b845a7d33299869 P 0f50df1481944fa5840dd2c440a5e747 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f50df1481944fa5840dd2c440a5e747" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 44355 } } peers { permanent_uuid: "78e4974170c34cf898aa560c9da7b69e" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 46023 } } peers { permanent_uuid: "b47206cd1be5454f97b1efada57a693e" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 43629 } }
I20260812 09:45:35.444922 22128 raft_consensus.cc:374] T 78166c891e074df39b845a7d33299869 P 78e4974170c34cf898aa560c9da7b69e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:45:35.444922 22127 raft_consensus.cc:374] T 78166c891e074df39b845a7d33299869 P b47206cd1be5454f97b1efada57a693e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:45:35.444989 22127 raft_consensus.cc:729] T 78166c891e074df39b845a7d33299869 P b47206cd1be5454f97b1efada57a693e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b47206cd1be5454f97b1efada57a693e, State: Initialized, Role: FOLLOWER
I20260812 09:45:35.444989 22128 raft_consensus.cc:729] T 78166c891e074df39b845a7d33299869 P 78e4974170c34cf898aa560c9da7b69e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78e4974170c34cf898aa560c9da7b69e, State: Initialized, Role: FOLLOWER
I20260812 09:45:35.445448 22128 consensus_queue.cc:260] T 78166c891e074df39b845a7d33299869 P 78e4974170c34cf898aa560c9da7b69e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f50df1481944fa5840dd2c440a5e747" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 44355 } } peers { permanent_uuid: "78e4974170c34cf898aa560c9da7b69e" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 46023 } } peers { permanent_uuid: "b47206cd1be5454f97b1efada57a693e" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 43629 } }
I20260812 09:45:35.445448 22127 consensus_queue.cc:260] T 78166c891e074df39b845a7d33299869 P b47206cd1be5454f97b1efada57a693e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f50df1481944fa5840dd2c440a5e747" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 44355 } } peers { permanent_uuid: "78e4974170c34cf898aa560c9da7b69e" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 46023 } } peers { permanent_uuid: "b47206cd1be5454f97b1efada57a693e" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 43629 } }
I20260812 09:45:35.445845 22129 ts_tablet_manager.cc:1428] T 78166c891e074df39b845a7d33299869 P 0f50df1481944fa5840dd2c440a5e747: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260812 09:45:35.446221 22088 heartbeater.cc:503] Master 127.13.99.62:41647 was elected leader, sending a full tablet report...
I20260812 09:45:35.446215 21700 heartbeater.cc:503] Master 127.13.99.62:41647 was elected leader, sending a full tablet report...
I20260812 09:45:35.446537 22128 ts_tablet_manager.cc:1428] T 78166c891e074df39b845a7d33299869 P 78e4974170c34cf898aa560c9da7b69e: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260812 09:45:35.447274 22127 ts_tablet_manager.cc:1428] T 78166c891e074df39b845a7d33299869 P b47206cd1be5454f97b1efada57a693e: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260812 09:45:35.447655 21895 heartbeater.cc:503] Master 127.13.99.62:41647 was elected leader, sending a full tablet report...
W20260812 09:45:35.516795 21705 tablet.cc:2369] T 78166c891e074df39b845a7d33299869 P 0f50df1481944fa5840dd2c440a5e747: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 09:45:35.589053 22134 raft_consensus.cc:482] T 78166c891e074df39b845a7d33299869 P 0f50df1481944fa5840dd2c440a5e747 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 09:45:35.589319 22134 raft_consensus.cc:504] T 78166c891e074df39b845a7d33299869 P 0f50df1481944fa5840dd2c440a5e747 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f50df1481944fa5840dd2c440a5e747" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 44355 } } peers { permanent_uuid: "78e4974170c34cf898aa560c9da7b69e" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 46023 } } peers { permanent_uuid: "b47206cd1be5454f97b1efada57a693e" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 43629 } }
I20260812 09:45:35.590576 22134 leader_election.cc:288] T 78166c891e074df39b845a7d33299869 P 0f50df1481944fa5840dd2c440a5e747 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78e4974170c34cf898aa560c9da7b69e (127.13.99.3:46023), b47206cd1be5454f97b1efada57a693e (127.13.99.2:43629)
I20260812 09:45:35.592886 22126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:35.591042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.1:49923 (local address 127.13.99.2:43629)
0812 09:45:35.591175 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:35.591180 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:35.591333 (+ 153us) server_negotiation.cc:407] Connection header received
0812 09:45:35.591438 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:35.591444 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:35.591493 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:35.591573 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:35.592413 (+ 840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:35.592417 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:35.592420 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 09:45:35.592433 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:35.592453 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 09:45:35.592467 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:35.592589 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:35.592667 (+ 78us) server_negotiation.cc:299] Negotiation successful
0812 09:45:35.592725 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260812 09:45:35.592912 22138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:35.590866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.2:43629 (local address 127.13.99.1:49923)
0812 09:45:35.591229 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:35.591242 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:35.591333 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:35.591622 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:35.591625 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:35.591639 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:45:35.592201 (+ 562us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 09:45:35.592204 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:45:35.592219 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 09:45:35.592257 (+ 38us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:45:35.592577 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:45:35.592581 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:45:35.592584 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 09:45:35.592675 (+ 91us) client_negotiation.cc:240] Negotiation successful
0812 09:45:35.592720 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":95,"threads_started":1}
I20260812 09:45:35.593071 22137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:35.590891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.3:46023 (local address 127.13.99.1:38309)
0812 09:45:35.591392 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:35.591408 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:35.591488 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:35.591767 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:35.591769 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:35.591781 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:45:35.592393 (+ 612us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 09:45:35.592395 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:45:35.592406 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 09:45:35.592432 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:45:35.592759 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:45:35.592762 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:45:35.592766 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 09:45:35.592881 (+ 115us) client_negotiation.cc:240] Negotiation successful
0812 09:45:35.592921 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":90,"threads_started":1}
I20260812 09:45:35.593073 22122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:35.590973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.1:38309 (local address 127.13.99.3:46023)
0812 09:45:35.591116 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:35.591121 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:35.591497 (+ 376us) server_negotiation.cc:407] Connection header received
0812 09:45:35.591604 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:35.591608 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:35.591648 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:35.591737 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:35.592602 (+ 865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:35.592608 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:35.592615 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 09:45:35.592631 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:35.592656 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 09:45:35.592666 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:35.592749 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:35.592888 (+ 139us) server_negotiation.cc:299] Negotiation successful
0812 09:45:35.592933 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 09:45:35.593669 21825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78166c891e074df39b845a7d33299869" candidate_uuid: "0f50df1481944fa5840dd2c440a5e747" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b47206cd1be5454f97b1efada57a693e" is_pre_election: true
I20260812 09:45:35.593654 22018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78166c891e074df39b845a7d33299869" candidate_uuid: "0f50df1481944fa5840dd2c440a5e747" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78e4974170c34cf898aa560c9da7b69e" is_pre_election: true
I20260812 09:45:35.594089 22018 raft_consensus.cc:2454] T 78166c891e074df39b845a7d33299869 P 78e4974170c34cf898aa560c9da7b69e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f50df1481944fa5840dd2c440a5e747 in term 0.
I20260812 09:45:35.594074 21825 raft_consensus.cc:2454] T 78166c891e074df39b845a7d33299869 P b47206cd1be5454f97b1efada57a693e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f50df1481944fa5840dd2c440a5e747 in term 0.
I20260812 09:45:35.594650 21530 leader_election.cc:302] T 78166c891e074df39b845a7d33299869 P 0f50df1481944fa5840dd2c440a5e747 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0f50df1481944fa5840dd2c440a5e747, 78e4974170c34cf898aa560c9da7b69e, b47206cd1be5454f97b1efada57a693e; no voters:
I20260812 09:45:35.594936 22134 raft_consensus.cc:2781] T 78166c891e074df39b845a7d33299869 P 0f50df1481944fa5840dd2c440a5e747 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 09:45:35.595029 22134 raft_consensus.cc:482] T 78166c891e074df39b845a7d33299869 P 0f50df1481944fa5840dd2c440a5e747 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 09:45:35.595074 22134 raft_consensus.cc:3037] T 78166c891e074df39b845a7d33299869 P 0f50df1481944fa5840dd2c440a5e747 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:45:35.596153 22134 raft_consensus.cc:504] T 78166c891e074df39b845a7d33299869 P 0f50df1481944fa5840dd2c440a5e747 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f50df1481944fa5840dd2c440a5e747" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 44355 } } peers { permanent_uuid: "78e4974170c34cf898aa560c9da7b69e" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 46023 } } peers { permanent_uuid: "b47206cd1be5454f97b1efada57a693e" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 43629 } }
I20260812 09:45:35.596527 22134 leader_election.cc:288] T 78166c891e074df39b845a7d33299869 P 0f50df1481944fa5840dd2c440a5e747 [CANDIDATE]: Term 1 election: Requested vote from peers 78e4974170c34cf898aa560c9da7b69e (127.13.99.3:46023), b47206cd1be5454f97b1efada57a693e (127.13.99.2:43629)
I20260812 09:45:35.596923 22018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78166c891e074df39b845a7d33299869" candidate_uuid: "0f50df1481944fa5840dd2c440a5e747" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78e4974170c34cf898aa560c9da7b69e"
I20260812 09:45:35.596920 21825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78166c891e074df39b845a7d33299869" candidate_uuid: "0f50df1481944fa5840dd2c440a5e747" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b47206cd1be5454f97b1efada57a693e"
I20260812 09:45:35.597056 22018 raft_consensus.cc:3037] T 78166c891e074df39b845a7d33299869 P 78e4974170c34cf898aa560c9da7b69e [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:45:35.597056 21825 raft_consensus.cc:3037] T 78166c891e074df39b845a7d33299869 P b47206cd1be5454f97b1efada57a693e [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:45:35.598372 21825 raft_consensus.cc:2454] T 78166c891e074df39b845a7d33299869 P b47206cd1be5454f97b1efada57a693e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f50df1481944fa5840dd2c440a5e747 in term 1.
I20260812 09:45:35.598376 22018 raft_consensus.cc:2454] T 78166c891e074df39b845a7d33299869 P 78e4974170c34cf898aa560c9da7b69e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f50df1481944fa5840dd2c440a5e747 in term 1.
I20260812 09:45:35.598685 21532 leader_election.cc:302] T 78166c891e074df39b845a7d33299869 P 0f50df1481944fa5840dd2c440a5e747 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f50df1481944fa5840dd2c440a5e747, 78e4974170c34cf898aa560c9da7b69e; no voters:
I20260812 09:45:35.598891 22134 raft_consensus.cc:2781] T 78166c891e074df39b845a7d33299869 P 0f50df1481944fa5840dd2c440a5e747 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 09:45:35.599123 22134 raft_consensus.cc:686] T 78166c891e074df39b845a7d33299869 P 0f50df1481944fa5840dd2c440a5e747 [term 1 LEADER]: Becoming Leader. State: Replica: 0f50df1481944fa5840dd2c440a5e747, State: Running, Role: LEADER
I20260812 09:45:35.599535 22134 consensus_queue.cc:237] T 78166c891e074df39b845a7d33299869 P 0f50df1481944fa5840dd2c440a5e747 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f50df1481944fa5840dd2c440a5e747" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 44355 } } peers { permanent_uuid: "78e4974170c34cf898aa560c9da7b69e" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 46023 } } peers { permanent_uuid: "b47206cd1be5454f97b1efada57a693e" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 43629 } }
I20260812 09:45:35.602890 21403 catalog_manager.cc:5495] T 78166c891e074df39b845a7d33299869 P 0f50df1481944fa5840dd2c440a5e747 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0f50df1481944fa5840dd2c440a5e747 (127.13.99.1). New cstate: current_term: 1 leader_uuid: "0f50df1481944fa5840dd2c440a5e747" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f50df1481944fa5840dd2c440a5e747" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 44355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78e4974170c34cf898aa560c9da7b69e" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 46023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b47206cd1be5454f97b1efada57a693e" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 43629 } health_report { overall_health: UNKNOWN } } }
I20260812 09:45:35.619366 22120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:35.617784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48868 (local address 127.13.99.1:44355)
0812 09:45:35.617929 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:35.617933 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:35.617949 (+ 16us) server_negotiation.cc:407] Connection header received
0812 09:45:35.618069 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:35.618071 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:35.618130 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:35.618221 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:35.618931 (+ 710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:35.618935 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:35.618940 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 09:45:35.618956 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:35.618973 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 09:45:35.618981 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:35.619077 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:35.619181 (+ 104us) server_negotiation.cc:299] Negotiation successful
0812 09:45:35.619222 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260812 09:45:35.625895 22018 raft_consensus.cc:1261] T 78166c891e074df39b845a7d33299869 P 78e4974170c34cf898aa560c9da7b69e [term 1 FOLLOWER]: Refusing update from remote peer 0f50df1481944fa5840dd2c440a5e747: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 09:45:35.625895 21825 raft_consensus.cc:1261] T 78166c891e074df39b845a7d33299869 P b47206cd1be5454f97b1efada57a693e [term 1 FOLLOWER]: Refusing update from remote peer 0f50df1481944fa5840dd2c440a5e747: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 09:45:35.626760 22134 consensus_queue.cc:1035] T 78166c891e074df39b845a7d33299869 P 0f50df1481944fa5840dd2c440a5e747 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e4974170c34cf898aa560c9da7b69e" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 46023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 09:45:35.627034 22140 consensus_queue.cc:1035] T 78166c891e074df39b845a7d33299869 P 0f50df1481944fa5840dd2c440a5e747 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b47206cd1be5454f97b1efada57a693e" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 43629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 09:45:35.627457 22090 tablet.cc:2369] T 78166c891e074df39b845a7d33299869 P 78e4974170c34cf898aa560c9da7b69e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 09:45:35.634413 22147 mvcc.cc:205] Tried to move back new op lower bound from 7317618424314437632 to 7317618424218542080. Current Snapshot: MvccSnapshot[applied={T|T < 7317618424314437632}]
I20260812 09:45:35.636346 22152 mvcc.cc:205] Tried to move back new op lower bound from 7317618424314437632 to 7317618424218542080. Current Snapshot: MvccSnapshot[applied={T|T < 7317618424314437632}]
I20260812 09:45:35.644491 21409 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59000:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 09:45:35.644677 21409 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59000:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 09:45:35.647161 21409 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e58f66a1a95947fb844738fbf18f09ab: Sending DeleteTablet for 3 replicas of tablet 78166c891e074df39b845a7d33299869
I20260812 09:45:35.647943 21987 tablet_service.cc:1514] Processing DeleteTablet for tablet 78166c891e074df39b845a7d33299869 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 09:45:35 UTC) from {username='slave'} at 127.0.0.1:53828
I20260812 09:45:35.648139 21594 tablet_service.cc:1514] Processing DeleteTablet for tablet 78166c891e074df39b845a7d33299869 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 09:45:35 UTC) from {username='slave'} at 127.0.0.1:48856
I20260812 09:45:35.648183 21792 tablet_service.cc:1514] Processing DeleteTablet for tablet 78166c891e074df39b845a7d33299869 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 09:45:35 UTC) from {username='slave'} at 127.0.0.1:44532
I20260812 09:45:35.648716 22163 tablet_replica.cc:330] T 78166c891e074df39b845a7d33299869 P b47206cd1be5454f97b1efada57a693e: stopping tablet replica
I20260812 09:45:35.648828 22162 tablet_replica.cc:330] T 78166c891e074df39b845a7d33299869 P 0f50df1481944fa5840dd2c440a5e747: stopping tablet replica
I20260812 09:45:35.648872 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 21509
I20260812 09:45:35.649063 22163 raft_consensus.cc:2229] T 78166c891e074df39b845a7d33299869 P b47206cd1be5454f97b1efada57a693e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 09:45:35.649140 22162 raft_consensus.cc:2229] T 78166c891e074df39b845a7d33299869 P 0f50df1481944fa5840dd2c440a5e747 [term 1 LEADER]: Raft consensus shutting down.
I20260812 09:45:35.649505 22163 raft_consensus.cc:2258] T 78166c891e074df39b845a7d33299869 P b47206cd1be5454f97b1efada57a693e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 09:45:35.662688 22161 tablet_replica.cc:330] T 78166c891e074df39b845a7d33299869 P 78e4974170c34cf898aa560c9da7b69e: stopping tablet replica
I20260812 09:45:35.663095 22161 raft_consensus.cc:2229] T 78166c891e074df39b845a7d33299869 P 78e4974170c34cf898aa560c9da7b69e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 09:45:35.663472 22161 raft_consensus.cc:2258] T 78166c891e074df39b845a7d33299869 P 78e4974170c34cf898aa560c9da7b69e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 09:45:35.664795 22163 ts_tablet_manager.cc:1905] T 78166c891e074df39b845a7d33299869 P b47206cd1be5454f97b1efada57a693e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 09:45:35.664880 22161 ts_tablet_manager.cc:1905] T 78166c891e074df39b845a7d33299869 P 78e4974170c34cf898aa560c9da7b69e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 09:45:35.667685 22161 ts_tablet_manager.cc:1918] T 78166c891e074df39b845a7d33299869 P 78e4974170c34cf898aa560c9da7b69e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 09:45:35.667685 22163 ts_tablet_manager.cc:1918] T 78166c891e074df39b845a7d33299869 P b47206cd1be5454f97b1efada57a693e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 09:45:35.667800 22161 log.cc:1196] T 78166c891e074df39b845a7d33299869 P 78e4974170c34cf898aa560c9da7b69e: Deleting WAL directory at /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786527905902985-13708-0/minicluster-data/ts-2/wal/wals/78166c891e074df39b845a7d33299869
I20260812 09:45:35.667800 22163 log.cc:1196] T 78166c891e074df39b845a7d33299869 P b47206cd1be5454f97b1efada57a693e: Deleting WAL directory at /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786527905902985-13708-0/minicluster-data/ts-1/wal/wals/78166c891e074df39b845a7d33299869
I20260812 09:45:35.668156 22163 ts_tablet_manager.cc:1939] T 78166c891e074df39b845a7d33299869 P b47206cd1be5454f97b1efada57a693e: Deleting consensus metadata
I20260812 09:45:35.668174 22161 ts_tablet_manager.cc:1939] T 78166c891e074df39b845a7d33299869 P 78e4974170c34cf898aa560c9da7b69e: Deleting consensus metadata
W20260812 09:45:35.669479 21385 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.99.1:44355 (error 108)
I20260812 09:45:35.669724 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 21707
I20260812 09:45:35.670046 21385 catalog_manager.cc:4841] TS b47206cd1be5454f97b1efada57a693e (127.13.99.2:43629): tablet 78166c891e074df39b845a7d33299869 (table test-table [id=1b52d5379b154361be585d7e59d52d9f]) successfully deleted
I20260812 09:45:35.672672 22121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:35.672312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.1:44355 (local address 127.0.0.1:48872)
0812 09:45:35.672435 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:35.672570 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.1:44355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260812 09:45:35.674052 21385 catalog_manager.cc:4584] TS 0f50df1481944fa5840dd2c440a5e747 (127.13.99.1:44355): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 78166c891e074df39b845a7d33299869: Network error: Client connection negotiation failed: client connection to 127.13.99.1:44355: connect: Connection refused (error 111)
I20260812 09:45:35.678346 21388 catalog_manager.cc:4841] TS 78e4974170c34cf898aa560c9da7b69e (127.13.99.3:46023): tablet 78166c891e074df39b845a7d33299869 (table test-table [id=1b52d5379b154361be585d7e59d52d9f]) successfully deleted
I20260812 09:45:35.685561 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 21900
I20260812 09:45:35.699564 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 21359
2026-08-12T09:45:35Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1155 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260812 09:45:35.729969 13708 mini_oidc.cc:152] Starting JWKS server
I20260812 09:45:35.732721 13708 webserver.cc:460] Webserver started at https://127.0.0.1:45387/ using document root <none> and password file <none>
I20260812 09:45:35.732913 13708 mini_oidc.cc:189] Starting OIDC Discovery server
I20260812 09:45:35.733196 13708 webserver.cc:460] Webserver started at http://127.0.0.1:45217/ using document root <none> and password file <none>
2026-08-12T09:45:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T09:45:35Z Disabled control of system clock
I20260812 09:45:35.743593 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786527905902985-13708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786527905902985-13708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786527905902985-13708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.62:41801
--webserver_interface=127.13.99.62
--webserver_port=0
--builtin_ntp_servers=127.13.99.20:36805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.99.62:41801
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45387/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786527905902985-13708-0/testchainca.cert with env {}
W20260812 09:45:35.856675 22174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:45:35.856947 22174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:45:35.857045 22174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:45:35.861100 22174 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 09:45:35.861155 22174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:45:35.861171 22174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:45:35.861187 22174 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 09:45:35.861203 22174 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260812 09:45:35.861264 22174 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260812 09:45:35.861284 22174 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45387/jwks.json
I20260812 09:45:35.866544 22174 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:36805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786527905902985-13708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786527905902985-13708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.99.62:41801
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.62:41801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45387/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.99.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786527905902985-13708-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22174
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786527905902985-13708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:45:35.867586 22174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:45:35.868709 22174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:45:35.877492 22186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:45:35.877707 22174 server_base.cc:1034] running on GCE node
W20260812 09:45:35.877553 22181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:35.877580 22180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:45:35.878458 22174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:45:35.879635 22174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:45:35.880822 22174 hybrid_clock.cc:648] HybridClock initialized: now 1786527935880793 us; error 40 us; skew 500 ppm
I20260812 09:45:35.882777 22174 webserver.cc:460] Webserver started at http://127.13.99.62:39423/ using document root <none> and password file <none>
I20260812 09:45:35.883374 22174 fs_manager.cc:359] Metadata directory not provided
I20260812 09:45:35.883472 22174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:45:35.883711 22174 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:45:35.885787 22174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786527905902985-13708-0/minicluster-data/master-0/data/instance:
uuid: "743059306f2a49f991e8f7164ee82ee9"
format_stamp: "Formatted at 2026-08-12 09:45:35 on dist-test-slave-jq4t"
server_key: "4b9f866038578846e94d1508fdf2860e"
server_key_iv: "37bf9d6365ae3aee24e14d7249007d53"
server_key_version: "encryptionkey@0"
I20260812 09:45:35.886389 22174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786527905902985-13708-0/minicluster-data/master-0/wal/instance:
uuid: "743059306f2a49f991e8f7164ee82ee9"
format_stamp: "Formatted at 2026-08-12 09:45:35 on dist-test-slave-jq4t"
server_key: "4b9f866038578846e94d1508fdf2860e"
server_key_iv: "37bf9d6365ae3aee24e14d7249007d53"
server_key_version: "encryptionkey@0"
I20260812 09:45:35.890153 22174 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260812 09:45:35.892787 22198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:35.894070 22174 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 09:45:35.894227 22174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786527905902985-13708-0/minicluster-data/master-0/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786527905902985-13708-0/minicluster-data/master-0/wal
uuid: "743059306f2a49f991e8f7164ee82ee9"
format_stamp: "Formatted at 2026-08-12 09:45:35 on dist-test-slave-jq4t"
server_key: "4b9f866038578846e94d1508fdf2860e"
server_key_iv: "37bf9d6365ae3aee24e14d7249007d53"
server_key_version: "encryptionkey@0"
I20260812 09:45:35.894346 22174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786527905902985-13708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786527905902985-13708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786527905902985-13708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:45:35.934084 22174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:45:35.934899 22174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:45:35.935130 22174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:45:35.943325 22174 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.62:41801
I20260812 09:45:35.943322 22273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.62:41801 every 8 connection(s)
I20260812 09:45:35.944415 22174 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
I20260812 09:45:35.947939 22275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:45:35.949971 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 22174
I20260812 09:45:35.950232 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786527905902985-13708-0/minicluster-data/master-0/wal/instance
I20260812 09:45:35.950587 13708 external_mini_cluster.cc:1428] Setting key 61b5ac4a127da26cc3673f22d7d8ac24
I20260812 09:45:35.954226 22275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 743059306f2a49f991e8f7164ee82ee9: Bootstrap starting.
I20260812 09:45:35.956442 22275 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 743059306f2a49f991e8f7164ee82ee9: Neither blocks nor log segments found. Creating new log.
I20260812 09:45:35.957209 22275 log.cc:824] T 00000000000000000000000000000000 P 743059306f2a49f991e8f7164ee82ee9: Log is configured to *not* fsync() on all Append() calls
I20260812 09:45:35.959528 22275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 743059306f2a49f991e8f7164ee82ee9: No bootstrap required, opened a new log
I20260812 09:45:35.960628 22281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:35.952125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38816 (local address 127.13.99.62:41801)
0812 09:45:35.952880 (+ 755us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:35.952898 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:35.952935 (+ 37us) server_negotiation.cc:407] Connection header received
0812 09:45:35.953593 (+ 658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:35.953612 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:35.954023 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:35.954369 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:35.954827 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:35.955785 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:35.956839 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:35.957313 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:35.957492 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:35.958464 (+ 972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:35.958471 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:35.958487 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 09:45:35.958982 (+ 495us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:35.959053 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 09:45:35.959080 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:35.959188 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:35.959369 (+ 181us) server_negotiation.cc:299] Negotiation successful
0812 09:45:35.959651 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":140,"threads_started":1}
I20260812 09:45:35.963708 13708 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 09:45:35.963946 22275 raft_consensus.cc:348] T 00000000000000000000000000000000 P 743059306f2a49f991e8f7164ee82ee9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "743059306f2a49f991e8f7164ee82ee9" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 41801 } }
I20260812 09:45:35.964195 22275 raft_consensus.cc:374] T 00000000000000000000000000000000 P 743059306f2a49f991e8f7164ee82ee9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:45:35.964285 22275 raft_consensus.cc:729] T 00000000000000000000000000000000 P 743059306f2a49f991e8f7164ee82ee9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 743059306f2a49f991e8f7164ee82ee9, State: Initialized, Role: FOLLOWER
I20260812 09:45:35.964835 22275 consensus_queue.cc:260] T 00000000000000000000000000000000 P 743059306f2a49f991e8f7164ee82ee9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "743059306f2a49f991e8f7164ee82ee9" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 41801 } }
I20260812 09:45:35.965014 22275 raft_consensus.cc:388] T 00000000000000000000000000000000 P 743059306f2a49f991e8f7164ee82ee9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 09:45:35.965085 22275 raft_consensus.cc:482] T 00000000000000000000000000000000 P 743059306f2a49f991e8f7164ee82ee9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 09:45:35.965184 22275 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 743059306f2a49f991e8f7164ee82ee9 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:45:35.966219 22275 raft_consensus.cc:504] T 00000000000000000000000000000000 P 743059306f2a49f991e8f7164ee82ee9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "743059306f2a49f991e8f7164ee82ee9" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 41801 } }
I20260812 09:45:35.966574 22275 leader_election.cc:302] T 00000000000000000000000000000000 P 743059306f2a49f991e8f7164ee82ee9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 743059306f2a49f991e8f7164ee82ee9; no voters:
I20260812 09:45:35.966953 22275 leader_election.cc:288] T 00000000000000000000000000000000 P 743059306f2a49f991e8f7164ee82ee9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 09:45:35.967082 22283 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 743059306f2a49f991e8f7164ee82ee9 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 09:45:35.967315 22283 raft_consensus.cc:686] T 00000000000000000000000000000000 P 743059306f2a49f991e8f7164ee82ee9 [term 1 LEADER]: Becoming Leader. State: Replica: 743059306f2a49f991e8f7164ee82ee9, State: Running, Role: LEADER
I20260812 09:45:35.967628 22283 consensus_queue.cc:237] T 00000000000000000000000000000000 P 743059306f2a49f991e8f7164ee82ee9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "743059306f2a49f991e8f7164ee82ee9" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 41801 } }
I20260812 09:45:35.968323 22275 sys_catalog.cc:564] T 00000000000000000000000000000000 P 743059306f2a49f991e8f7164ee82ee9 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 09:45:35.969631 22285 sys_catalog.cc:455] T 00000000000000000000000000000000 P 743059306f2a49f991e8f7164ee82ee9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "743059306f2a49f991e8f7164ee82ee9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "743059306f2a49f991e8f7164ee82ee9" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 41801 } } }
I20260812 09:45:35.969761 22285 sys_catalog.cc:458] T 00000000000000000000000000000000 P 743059306f2a49f991e8f7164ee82ee9 [sys.catalog]: This master's current role is: LEADER
I20260812 09:45:35.969659 22287 sys_catalog.cc:455] T 00000000000000000000000000000000 P 743059306f2a49f991e8f7164ee82ee9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 743059306f2a49f991e8f7164ee82ee9. Latest consensus state: current_term: 1 leader_uuid: "743059306f2a49f991e8f7164ee82ee9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "743059306f2a49f991e8f7164ee82ee9" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 41801 } } }
I20260812 09:45:35.969933 22287 sys_catalog.cc:458] T 00000000000000000000000000000000 P 743059306f2a49f991e8f7164ee82ee9 [sys.catalog]: This master's current role is: LEADER
I20260812 09:45:35.970209 22299 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 09:45:35.973023 22299 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 09:45:35.978861 22299 catalog_manager.cc:1347] Generated new cluster ID: 5b8b547f17d04b159b74b1eaddb306eb
I20260812 09:45:35.978955 22299 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 09:45:36.000429 22299 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 09:45:36.001415 22299 catalog_manager.cc:1504] Loading token signing keys...
I20260812 09:45:36.014178 22299 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 743059306f2a49f991e8f7164ee82ee9: Generated new TSK 0
I20260812 09:45:36.015070 22299 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260812 09:45:36.052595 22319 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260812 09:45:36.052728 22319 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260812 09:45:36.053480 22281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:36.027440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38826 (local address 127.13.99.62:41801)
0812 09:45:36.027590 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:36.027595 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:36.027811 (+ 216us) server_negotiation.cc:407] Connection header received
0812 09:45:36.028028 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:36.028031 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:36.028080 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:36.028167 (+ 87us) server_negotiation.cc:226] Negotiated authn=JWT
0812 09:45:36.028467 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:36.029080 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:36.030345 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:36.030591 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:36.030683 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:36.031102 (+ 419us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0812 09:45:36.053011 (+ 21909us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45387/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0812 09:45:36.053323 (+ 312us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38826: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45387/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":61}
W20260812 09:45:36.053571 22281 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38826: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45387/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260812 09:45:36.054971 22318 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 09:45:36.027360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:41801 (local address 127.0.0.1:38826)
0812 09:45:36.027715 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:36.027732 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:36.027871 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:36.028205 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:36.028209 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:36.028217 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0812 09:45:36.028350 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:36.028356 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:36.029196 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:36.029200 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:36.030204 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:36.030216 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:36.030712 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:36.030716 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:36.030829 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:36.030887 (+ 58us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0812 09:45:36.053320 (+ 22433us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45387/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0812 09:45:36.053605 (+ 285us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.13.99.62:41801: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45387/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":129,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.13.99.62:41801: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45387/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260812 09:45:36.055819 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 22174
2026-08-12T09:45:36Z chronyd exiting
I20260812 09:45:36.098407 13708 test_util.cc:175] -----------------------------------------------
I20260812 09:45:36.098531 13708 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786527905902985-13708-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (370 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260812 09:45:36.099742 13708 mini_oidc.cc:152] Starting JWKS server
I20260812 09:45:36.100768 13708 webserver.cc:460] Webserver started at https://127.0.0.1:33577/ using document root <none> and password file <none>
I20260812 09:45:36.100901 13708 mini_oidc.cc:189] Starting OIDC Discovery server
I20260812 09:45:36.101184 13708 webserver.cc:460] Webserver started at http://127.0.0.1:32777/ using document root <none> and password file <none>
2026-08-12T09:45:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T09:45:36Z Disabled control of system clock
I20260812 09:45:36.111857 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786527905902985-13708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786527905902985-13708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786527905902985-13708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.62:37853
--webserver_interface=127.13.99.62
--webserver_port=0
--builtin_ntp_servers=127.13.99.20:43175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.99.62:37853
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33577/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786527905902985-13708-0/testchainca.cert with env {}
W20260812 09:45:36.230319 22327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:45:36.230579 22327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:45:36.230618 22327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:45:36.234512 22327 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 09:45:36.234561 22327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:45:36.234576 22327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:45:36.234592 22327 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 09:45:36.234607 22327 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260812 09:45:36.234622 22327 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260812 09:45:36.234635 22327 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33577/jwks.json
I20260812 09:45:36.239830 22327 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:43175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786527905902985-13708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786527905902985-13708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.99.62:37853
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.62:37853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33577/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.99.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786527905902985-13708-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22327
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786527905902985-13708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:45:36.240823 22327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:45:36.241775 22327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:45:36.250279 22341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:45:36.250375 22327 server_base.cc:1034] running on GCE node
W20260812 09:45:36.250319 22338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:36.250329 22335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:45:36.251086 22327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:45:36.252233 22327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:45:36.253430 22327 hybrid_clock.cc:648] HybridClock initialized: now 1786527936253399 us; error 53 us; skew 500 ppm
I20260812 09:45:36.255342 22327 webserver.cc:460] Webserver started at http://127.13.99.62:33511/ using document root <none> and password file <none>
I20260812 09:45:36.255937 22327 fs_manager.cc:359] Metadata directory not provided
I20260812 09:45:36.256033 22327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:45:36.256268 22327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:45:36.258332 22327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786527905902985-13708-0/minicluster-data/master-0/data/instance:
uuid: "4c26fc544b4947fd845695393f7e1b41"
format_stamp: "Formatted at 2026-08-12 09:45:36 on dist-test-slave-jq4t"
server_key: "d720d55bb1d3f2764b5377cd321a10ea"
server_key_iv: "0748cb003bd3018632f7420cafb277c9"
server_key_version: "encryptionkey@0"
I20260812 09:45:36.258913 22327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786527905902985-13708-0/minicluster-data/master-0/wal/instance:
uuid: "4c26fc544b4947fd845695393f7e1b41"
format_stamp: "Formatted at 2026-08-12 09:45:36 on dist-test-slave-jq4t"
server_key: "d720d55bb1d3f2764b5377cd321a10ea"
server_key_iv: "0748cb003bd3018632f7420cafb277c9"
server_key_version: "encryptionkey@0"
I20260812 09:45:36.262580 22327 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260812 09:45:36.264935 22348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:36.266057 22327 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 09:45:36.266206 22327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786527905902985-13708-0/minicluster-data/master-0/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786527905902985-13708-0/minicluster-data/master-0/wal
uuid: "4c26fc544b4947fd845695393f7e1b41"
format_stamp: "Formatted at 2026-08-12 09:45:36 on dist-test-slave-jq4t"
server_key: "d720d55bb1d3f2764b5377cd321a10ea"
server_key_iv: "0748cb003bd3018632f7420cafb277c9"
server_key_version: "encryptionkey@0"
I20260812 09:45:36.266316 22327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786527905902985-13708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786527905902985-13708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786527905902985-13708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:45:36.282697 22327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:45:36.283466 22327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:45:36.283684 22327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:45:36.291188 22327 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.62:37853
I20260812 09:45:36.291203 22422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.62:37853 every 8 connection(s)
I20260812 09:45:36.292177 22327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
I20260812 09:45:36.295572 22424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:45:36.297989 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 22327
I20260812 09:45:36.298120 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786527905902985-13708-0/minicluster-data/master-0/wal/instance
I20260812 09:45:36.298417 13708 external_mini_cluster.cc:1428] Setting key fd0aff719bf9d85c61795de718303ac0
I20260812 09:45:36.302799 22424 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c26fc544b4947fd845695393f7e1b41: Bootstrap starting.
I20260812 09:45:36.305114 22424 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c26fc544b4947fd845695393f7e1b41: Neither blocks nor log segments found. Creating new log.
I20260812 09:45:36.306016 22424 log.cc:824] T 00000000000000000000000000000000 P 4c26fc544b4947fd845695393f7e1b41: Log is configured to *not* fsync() on all Append() calls
I20260812 09:45:36.308076 22427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:36.299756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40950 (local address 127.13.99.62:37853)
0812 09:45:36.300366 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:36.300381 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:36.300431 (+ 50us) server_negotiation.cc:407] Connection header received
0812 09:45:36.301159 (+ 728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:36.301176 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:36.301652 (+ 476us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:36.301979 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:36.302423 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:36.303470 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:36.304369 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:36.304911 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:36.305097 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:36.306034 (+ 937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:36.306039 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:36.306053 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 09:45:36.306551 (+ 498us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:36.306625 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 09:45:36.306654 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:36.306762 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:36.307020 (+ 258us) server_negotiation.cc:299] Negotiation successful
0812 09:45:36.307351 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":108,"threads_started":1}
I20260812 09:45:36.308084 22424 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c26fc544b4947fd845695393f7e1b41: No bootstrap required, opened a new log
I20260812 09:45:36.311146 13708 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 09:45:36.312413 22424 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c26fc544b4947fd845695393f7e1b41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c26fc544b4947fd845695393f7e1b41" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 37853 } }
I20260812 09:45:36.312714 22424 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c26fc544b4947fd845695393f7e1b41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:45:36.312795 22424 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c26fc544b4947fd845695393f7e1b41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c26fc544b4947fd845695393f7e1b41, State: Initialized, Role: FOLLOWER
I20260812 09:45:36.313233 22424 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c26fc544b4947fd845695393f7e1b41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c26fc544b4947fd845695393f7e1b41" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 37853 } }
I20260812 09:45:36.313385 22424 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c26fc544b4947fd845695393f7e1b41 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 09:45:36.313473 22424 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c26fc544b4947fd845695393f7e1b41 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 09:45:36.313601 22424 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c26fc544b4947fd845695393f7e1b41 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:45:36.314539 22424 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c26fc544b4947fd845695393f7e1b41 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c26fc544b4947fd845695393f7e1b41" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 37853 } }
I20260812 09:45:36.314915 22424 leader_election.cc:302] T 00000000000000000000000000000000 P 4c26fc544b4947fd845695393f7e1b41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c26fc544b4947fd845695393f7e1b41; no voters:
I20260812 09:45:36.315284 22424 leader_election.cc:288] T 00000000000000000000000000000000 P 4c26fc544b4947fd845695393f7e1b41 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 09:45:36.315392 22430 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c26fc544b4947fd845695393f7e1b41 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 09:45:36.315649 22430 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c26fc544b4947fd845695393f7e1b41 [term 1 LEADER]: Becoming Leader. State: Replica: 4c26fc544b4947fd845695393f7e1b41, State: Running, Role: LEADER
I20260812 09:45:36.315945 22430 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c26fc544b4947fd845695393f7e1b41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c26fc544b4947fd845695393f7e1b41" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 37853 } }
I20260812 09:45:36.316568 22424 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c26fc544b4947fd845695393f7e1b41 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 09:45:36.317678 22433 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c26fc544b4947fd845695393f7e1b41 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c26fc544b4947fd845695393f7e1b41. Latest consensus state: current_term: 1 leader_uuid: "4c26fc544b4947fd845695393f7e1b41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c26fc544b4947fd845695393f7e1b41" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 37853 } } }
I20260812 09:45:36.317768 22431 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c26fc544b4947fd845695393f7e1b41 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c26fc544b4947fd845695393f7e1b41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c26fc544b4947fd845695393f7e1b41" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 37853 } } }
I20260812 09:45:36.317829 22433 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c26fc544b4947fd845695393f7e1b41 [sys.catalog]: This master's current role is: LEADER
I20260812 09:45:36.317829 22431 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c26fc544b4947fd845695393f7e1b41 [sys.catalog]: This master's current role is: LEADER
I20260812 09:45:36.318245 22442 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 09:45:36.321302 22442 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 09:45:36.326790 22442 catalog_manager.cc:1347] Generated new cluster ID: 99885c4a547c4e0797f341d756c5d87f
I20260812 09:45:36.326869 22442 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 09:45:36.335500 22442 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 09:45:36.336517 22442 catalog_manager.cc:1504] Loading token signing keys...
I20260812 09:45:36.342162 22442 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c26fc544b4947fd845695393f7e1b41: Generated new TSK 0
I20260812 09:45:36.342888 22442 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260812 09:45:36.377949 22462 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260812 09:45:36.378052 22462 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260812 09:45:36.378736 22427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:36.354816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40962 (local address 127.13.99.62:37853)
0812 09:45:36.354995 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:36.355001 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:36.355194 (+ 193us) server_negotiation.cc:407] Connection header received
0812 09:45:36.355330 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:36.355333 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:36.355386 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:36.355480 (+ 94us) server_negotiation.cc:226] Negotiated authn=JWT
0812 09:45:36.355790 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:36.356371 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:36.357688 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:36.357944 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:36.358024 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:36.358368 (+ 344us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0812 09:45:36.378325 (+ 19957us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33577/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0812 09:45:36.378601 (+ 276us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40962: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33577/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":71}
W20260812 09:45:36.378834 22427 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40962: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33577/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260812 09:45:36.378945 22460 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 09:45:36.354722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:37853 (local address 127.0.0.1:40962)
0812 09:45:36.355055 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:36.355071 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:36.355194 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:36.355505 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:36.355508 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:36.355517 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0812 09:45:36.355649 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:36.355656 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:36.356533 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:36.356537 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:36.357556 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:36.357567 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:36.358062 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:36.358066 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:36.358164 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:36.358194 (+ 30us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0812 09:45:36.378559 (+ 20365us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33577/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0812 09:45:36.378757 (+ 198us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.13.99.62:37853: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33577/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":126,"threads_started":1}
I20260812 09:45:36.379698 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 22327
2026-08-12T09:45:36Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (331 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260812 09:45:36.431293 13708 mini_oidc.cc:152] Starting JWKS server
I20260812 09:45:36.432278 13708 webserver.cc:460] Webserver started at https://127.0.0.1:33161/ using document root <none> and password file <none>
I20260812 09:45:36.432427 13708 mini_oidc.cc:189] Starting OIDC Discovery server
I20260812 09:45:36.432801 13708 webserver.cc:460] Webserver started at http://127.0.0.1:41085/ using document root <none> and password file <none>
2026-08-12T09:45:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T09:45:36Z Disabled control of system clock
I20260812 09:45:36.443804 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786527905902985-13708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786527905902985-13708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786527905902985-13708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.62:34553
--webserver_interface=127.13.99.62
--webserver_port=0
--builtin_ntp_servers=127.13.99.20:35367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.99.62:34553
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33161/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260812 09:45:36.560225 22469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:45:36.560554 22469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:45:36.560663 22469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:45:36.564589 22469 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 09:45:36.564670 22469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:45:36.564700 22469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:45:36.564738 22469 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 09:45:36.564770 22469 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260812 09:45:36.564802 22469 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260812 09:45:36.564834 22469 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33161/jwks.json
I20260812 09:45:36.569574 22469 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:35367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786527905902985-13708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786527905902985-13708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.99.62:34553
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.62:34553
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33161/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.99.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22469
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786527905902985-13708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:45:36.570618 22469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:45:36.572017 22469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:45:36.580761 22476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:36.580780 22479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:36.580780 22475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:45:36.580999 22469 server_base.cc:1034] running on GCE node
I20260812 09:45:36.581555 22469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:45:36.582568 22469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:45:36.583736 22469 hybrid_clock.cc:648] HybridClock initialized: now 1786527936583720 us; error 34 us; skew 500 ppm
I20260812 09:45:36.585636 22469 webserver.cc:460] Webserver started at http://127.13.99.62:40675/ using document root <none> and password file <none>
I20260812 09:45:36.586218 22469 fs_manager.cc:359] Metadata directory not provided
I20260812 09:45:36.586280 22469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:45:36.586546 22469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:45:36.588593 22469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786527905902985-13708-0/minicluster-data/master-0/data/instance:
uuid: "979a5602217f4a4c99a585afaa1a5fd2"
format_stamp: "Formatted at 2026-08-12 09:45:36 on dist-test-slave-jq4t"
server_key: "8a4f72a31acc2a7e1b95642c15ef5da8"
server_key_iv: "335fad415ad807e44a4a805599ecea53"
server_key_version: "encryptionkey@0"
I20260812 09:45:36.589159 22469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786527905902985-13708-0/minicluster-data/master-0/wal/instance:
uuid: "979a5602217f4a4c99a585afaa1a5fd2"
format_stamp: "Formatted at 2026-08-12 09:45:36 on dist-test-slave-jq4t"
server_key: "8a4f72a31acc2a7e1b95642c15ef5da8"
server_key_iv: "335fad415ad807e44a4a805599ecea53"
server_key_version: "encryptionkey@0"
I20260812 09:45:36.592710 22469 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 09:45:36.595113 22491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:36.596299 22469 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 09:45:36.596441 22469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786527905902985-13708-0/minicluster-data/master-0/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786527905902985-13708-0/minicluster-data/master-0/wal
uuid: "979a5602217f4a4c99a585afaa1a5fd2"
format_stamp: "Formatted at 2026-08-12 09:45:36 on dist-test-slave-jq4t"
server_key: "8a4f72a31acc2a7e1b95642c15ef5da8"
server_key_iv: "335fad415ad807e44a4a805599ecea53"
server_key_version: "encryptionkey@0"
I20260812 09:45:36.596598 22469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786527905902985-13708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786527905902985-13708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786527905902985-13708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:45:36.607851 22469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:45:36.608569 22469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:45:36.608764 22469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:45:36.615819 22469 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.62:34553
I20260812 09:45:36.615844 22572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.62:34553 every 8 connection(s)
I20260812 09:45:36.616834 22469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
I20260812 09:45:36.620321 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 22469
I20260812 09:45:36.620455 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786527905902985-13708-0/minicluster-data/master-0/wal/instance
I20260812 09:45:36.620618 22574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:45:36.620780 13708 external_mini_cluster.cc:1428] Setting key a065588930e6005431bf4e063fc57782
I20260812 09:45:36.627209 22574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 979a5602217f4a4c99a585afaa1a5fd2: Bootstrap starting.
I20260812 09:45:36.629724 22574 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 979a5602217f4a4c99a585afaa1a5fd2: Neither blocks nor log segments found. Creating new log.
I20260812 09:45:36.630111 22578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:36.622188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33332 (local address 127.13.99.62:34553)
0812 09:45:36.622844 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:36.622884 (+ 40us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:36.622921 (+ 37us) server_negotiation.cc:407] Connection header received
0812 09:45:36.623538 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:36.623555 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:36.623944 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:36.624285 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:36.624754 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:36.625760 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:36.626682 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:36.627163 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:36.627339 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:36.628261 (+ 922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:36.628266 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:36.628278 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 09:45:36.628719 (+ 441us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:36.628783 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 09:45:36.628805 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:36.628923 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:36.629094 (+ 171us) server_negotiation.cc:299] Negotiation successful
0812 09:45:36.629352 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":135,"threads_started":1}
I20260812 09:45:36.630638 22574 log.cc:824] T 00000000000000000000000000000000 P 979a5602217f4a4c99a585afaa1a5fd2: Log is configured to *not* fsync() on all Append() calls
I20260812 09:45:36.632826 22574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 979a5602217f4a4c99a585afaa1a5fd2: No bootstrap required, opened a new log
I20260812 09:45:36.633092 13708 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 09:45:36.636595 22574 raft_consensus.cc:348] T 00000000000000000000000000000000 P 979a5602217f4a4c99a585afaa1a5fd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "979a5602217f4a4c99a585afaa1a5fd2" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 34553 } }
I20260812 09:45:36.636814 22574 raft_consensus.cc:374] T 00000000000000000000000000000000 P 979a5602217f4a4c99a585afaa1a5fd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:45:36.636850 22574 raft_consensus.cc:729] T 00000000000000000000000000000000 P 979a5602217f4a4c99a585afaa1a5fd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 979a5602217f4a4c99a585afaa1a5fd2, State: Initialized, Role: FOLLOWER
I20260812 09:45:36.637218 22574 consensus_queue.cc:260] T 00000000000000000000000000000000 P 979a5602217f4a4c99a585afaa1a5fd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "979a5602217f4a4c99a585afaa1a5fd2" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 34553 } }
I20260812 09:45:36.637326 22574 raft_consensus.cc:388] T 00000000000000000000000000000000 P 979a5602217f4a4c99a585afaa1a5fd2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 09:45:36.637372 22574 raft_consensus.cc:482] T 00000000000000000000000000000000 P 979a5602217f4a4c99a585afaa1a5fd2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 09:45:36.637444 22574 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 979a5602217f4a4c99a585afaa1a5fd2 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:45:36.638521 22574 raft_consensus.cc:504] T 00000000000000000000000000000000 P 979a5602217f4a4c99a585afaa1a5fd2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "979a5602217f4a4c99a585afaa1a5fd2" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 34553 } }
I20260812 09:45:36.638845 22574 leader_election.cc:302] T 00000000000000000000000000000000 P 979a5602217f4a4c99a585afaa1a5fd2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 979a5602217f4a4c99a585afaa1a5fd2; no voters:
I20260812 09:45:36.639125 22574 leader_election.cc:288] T 00000000000000000000000000000000 P 979a5602217f4a4c99a585afaa1a5fd2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 09:45:36.639298 22580 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 979a5602217f4a4c99a585afaa1a5fd2 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 09:45:36.639566 22580 raft_consensus.cc:686] T 00000000000000000000000000000000 P 979a5602217f4a4c99a585afaa1a5fd2 [term 1 LEADER]: Becoming Leader. State: Replica: 979a5602217f4a4c99a585afaa1a5fd2, State: Running, Role: LEADER
I20260812 09:45:36.639954 22580 consensus_queue.cc:237] T 00000000000000000000000000000000 P 979a5602217f4a4c99a585afaa1a5fd2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "979a5602217f4a4c99a585afaa1a5fd2" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 34553 } }
I20260812 09:45:36.640621 22574 sys_catalog.cc:564] T 00000000000000000000000000000000 P 979a5602217f4a4c99a585afaa1a5fd2 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 09:45:36.641974 22582 sys_catalog.cc:455] T 00000000000000000000000000000000 P 979a5602217f4a4c99a585afaa1a5fd2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 979a5602217f4a4c99a585afaa1a5fd2. Latest consensus state: current_term: 1 leader_uuid: "979a5602217f4a4c99a585afaa1a5fd2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "979a5602217f4a4c99a585afaa1a5fd2" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 34553 } } }
I20260812 09:45:36.642015 22581 sys_catalog.cc:455] T 00000000000000000000000000000000 P 979a5602217f4a4c99a585afaa1a5fd2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "979a5602217f4a4c99a585afaa1a5fd2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "979a5602217f4a4c99a585afaa1a5fd2" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 34553 } } }
I20260812 09:45:36.642184 22581 sys_catalog.cc:458] T 00000000000000000000000000000000 P 979a5602217f4a4c99a585afaa1a5fd2 [sys.catalog]: This master's current role is: LEADER
I20260812 09:45:36.642179 22582 sys_catalog.cc:458] T 00000000000000000000000000000000 P 979a5602217f4a4c99a585afaa1a5fd2 [sys.catalog]: This master's current role is: LEADER
I20260812 09:45:36.642732 22596 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 09:45:36.645561 22596 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 09:45:36.650624 22596 catalog_manager.cc:1347] Generated new cluster ID: 490968b4a09a46d9a5d668e1500c6c1a
I20260812 09:45:36.650691 22596 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 09:45:36.657603 22596 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 09:45:36.658526 22596 catalog_manager.cc:1504] Loading token signing keys...
I20260812 09:45:36.664609 22596 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 979a5602217f4a4c99a585afaa1a5fd2: Generated new TSK 0
I20260812 09:45:36.665318 22596 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 09:45:36.685976 22578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:36.680848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33342 (local address 127.13.99.62:34553)
0812 09:45:36.680983 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:36.680988 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:36.681217 (+ 229us) server_negotiation.cc:407] Connection header received
0812 09:45:36.681326 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:36.681328 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:36.681386 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:36.681484 (+ 98us) server_negotiation.cc:226] Negotiated authn=JWT
0812 09:45:36.681770 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:36.682309 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:36.683603 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:36.683815 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:36.683923 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:36.684307 (+ 384us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0812 09:45:36.685602 (+ 1295us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33161/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0812 09:45:36.685865 (+ 263us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33342: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33161/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":55}
W20260812 09:45:36.686069 22578 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33342: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33161/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260812 09:45:36.688850 22609 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 09:45:36.680701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:34553 (local address 127.0.0.1:33342)
0812 09:45:36.681054 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:36.681070 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:36.681195 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:36.681527 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:36.681530 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:36.681538 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0812 09:45:36.681654 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:36.681662 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:36.682440 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:36.682444 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:36.683459 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:36.683470 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:36.683954 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:36.683959 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:36.684050 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:36.684080 (+ 30us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0812 09:45:36.685768 (+ 1688us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33161/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0812 09:45:36.688717 (+ 2949us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.13.99.62:34553: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33161/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":258,"mutex_wait_us":2776,"thread_start_us":113,"threads_started":1}
I20260812 09:45:36.689445 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 22469
E20260812 09:45:36.711609 22611 webserver.cc:533] Webserver: sslize failed:
E20260812 09:45:36.711696 22611 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-08-12T09:45:36Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (311 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-12T09:45:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T09:45:36Z Disabled control of system clock
I20260812 09:45:36.752653 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786527905902985-13708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786527905902985-13708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786527905902985-13708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.62:43059
--webserver_interface=127.13.99.62
--webserver_port=0
--builtin_ntp_servers=127.13.99.20:39943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.99.62:43059
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786527905902985-13708-0/testchainca.cert with env {}
W20260812 09:45:36.868156 22618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:45:36.868402 22618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:45:36.868440 22618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:45:36.872442 22618 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 09:45:36.872524 22618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:45:36.872541 22618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:45:36.872557 22618 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 09:45:36.872572 22618 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260812 09:45:36.872588 22618 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260812 09:45:36.872601 22618 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260812 09:45:36.878055 22618 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:39943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786527905902985-13708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786527905902985-13708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.99.62:43059
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.62:43059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.99.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786527905902985-13708-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22618
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786527905902985-13708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:45:36.879172 22618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:45:36.880203 22618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:45:36.888897 22628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:36.888864 22624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:36.888871 22626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:45:36.889278 22618 server_base.cc:1034] running on GCE node
I20260812 09:45:36.889873 22618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:45:36.890960 22618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:45:36.892123 22618 hybrid_clock.cc:648] HybridClock initialized: now 1786527936892097 us; error 40 us; skew 500 ppm
I20260812 09:45:36.893999 22618 webserver.cc:460] Webserver started at http://127.13.99.62:34361/ using document root <none> and password file <none>
I20260812 09:45:36.894578 22618 fs_manager.cc:359] Metadata directory not provided
I20260812 09:45:36.894640 22618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:45:36.894901 22618 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:45:36.896974 22618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786527905902985-13708-0/minicluster-data/master-0/data/instance:
uuid: "789e1f0ec7034fd9ba76b038312fe76f"
format_stamp: "Formatted at 2026-08-12 09:45:36 on dist-test-slave-jq4t"
server_key: "80681ac4d9d00ba3ee062ae9063cb3f3"
server_key_iv: "f152a068c80dcab38bdb1d7bf5221259"
server_key_version: "encryptionkey@0"
I20260812 09:45:36.897535 22618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786527905902985-13708-0/minicluster-data/master-0/wal/instance:
uuid: "789e1f0ec7034fd9ba76b038312fe76f"
format_stamp: "Formatted at 2026-08-12 09:45:36 on dist-test-slave-jq4t"
server_key: "80681ac4d9d00ba3ee062ae9063cb3f3"
server_key_iv: "f152a068c80dcab38bdb1d7bf5221259"
server_key_version: "encryptionkey@0"
I20260812 09:45:36.901264 22618 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260812 09:45:36.903644 22639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:36.904976 22618 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260812 09:45:36.905128 22618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786527905902985-13708-0/minicluster-data/master-0/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786527905902985-13708-0/minicluster-data/master-0/wal
uuid: "789e1f0ec7034fd9ba76b038312fe76f"
format_stamp: "Formatted at 2026-08-12 09:45:36 on dist-test-slave-jq4t"
server_key: "80681ac4d9d00ba3ee062ae9063cb3f3"
server_key_iv: "f152a068c80dcab38bdb1d7bf5221259"
server_key_version: "encryptionkey@0"
I20260812 09:45:36.905257 22618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786527905902985-13708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786527905902985-13708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786527905902985-13708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:45:36.923050 22618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:45:36.923678 22618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:45:36.923828 22618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:45:36.931347 22618 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.62:43059
I20260812 09:45:36.931363 22713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.62:43059 every 8 connection(s)
I20260812 09:45:36.932340 22618 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
I20260812 09:45:36.935293 22714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:45:36.939061 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 22618
I20260812 09:45:36.939185 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786527905902985-13708-0/minicluster-data/master-0/wal/instance
I20260812 09:45:36.939515 13708 external_mini_cluster.cc:1428] Setting key aa4230eef3fa2189c42c00c32c1699d9
I20260812 09:45:36.941391 22714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 789e1f0ec7034fd9ba76b038312fe76f: Bootstrap starting.
I20260812 09:45:36.943660 22714 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 789e1f0ec7034fd9ba76b038312fe76f: Neither blocks nor log segments found. Creating new log.
I20260812 09:45:36.944413 22714 log.cc:824] T 00000000000000000000000000000000 P 789e1f0ec7034fd9ba76b038312fe76f: Log is configured to *not* fsync() on all Append() calls
I20260812 09:45:36.946316 22714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 789e1f0ec7034fd9ba76b038312fe76f: No bootstrap required, opened a new log
I20260812 09:45:36.949198 22718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:36.941028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41022 (local address 127.13.99.62:43059)
0812 09:45:36.941770 (+ 742us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:36.941787 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:36.941825 (+ 38us) server_negotiation.cc:407] Connection header received
0812 09:45:36.942496 (+ 671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:36.942513 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:36.942975 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:36.943331 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:36.943858 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:36.944968 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:36.945860 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:36.946340 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:36.946531 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:36.947437 (+ 906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:36.947442 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:36.947455 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 09:45:36.947836 (+ 381us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:36.947892 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 09:45:36.947912 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:36.948033 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:36.948152 (+ 119us) server_negotiation.cc:299] Negotiation successful
0812 09:45:36.948409 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":139,"threads_started":1}
I20260812 09:45:36.950101 22714 raft_consensus.cc:348] T 00000000000000000000000000000000 P 789e1f0ec7034fd9ba76b038312fe76f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "789e1f0ec7034fd9ba76b038312fe76f" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 43059 } }
I20260812 09:45:36.950342 22714 raft_consensus.cc:374] T 00000000000000000000000000000000 P 789e1f0ec7034fd9ba76b038312fe76f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:45:36.950439 22714 raft_consensus.cc:729] T 00000000000000000000000000000000 P 789e1f0ec7034fd9ba76b038312fe76f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 789e1f0ec7034fd9ba76b038312fe76f, State: Initialized, Role: FOLLOWER
I20260812 09:45:36.950888 22714 consensus_queue.cc:260] T 00000000000000000000000000000000 P 789e1f0ec7034fd9ba76b038312fe76f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "789e1f0ec7034fd9ba76b038312fe76f" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 43059 } }
I20260812 09:45:36.951040 22714 raft_consensus.cc:388] T 00000000000000000000000000000000 P 789e1f0ec7034fd9ba76b038312fe76f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 09:45:36.951116 22714 raft_consensus.cc:482] T 00000000000000000000000000000000 P 789e1f0ec7034fd9ba76b038312fe76f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 09:45:36.951226 22714 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 789e1f0ec7034fd9ba76b038312fe76f [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:45:36.952229 13708 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 09:45:36.952194 22714 raft_consensus.cc:504] T 00000000000000000000000000000000 P 789e1f0ec7034fd9ba76b038312fe76f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "789e1f0ec7034fd9ba76b038312fe76f" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 43059 } }
I20260812 09:45:36.952569 22714 leader_election.cc:302] T 00000000000000000000000000000000 P 789e1f0ec7034fd9ba76b038312fe76f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 789e1f0ec7034fd9ba76b038312fe76f; no voters:
I20260812 09:45:36.952909 22714 leader_election.cc:288] T 00000000000000000000000000000000 P 789e1f0ec7034fd9ba76b038312fe76f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 09:45:36.953008 22723 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 789e1f0ec7034fd9ba76b038312fe76f [term 1 FOLLOWER]: Leader election won for term 1
I20260812 09:45:36.953271 22723 raft_consensus.cc:686] T 00000000000000000000000000000000 P 789e1f0ec7034fd9ba76b038312fe76f [term 1 LEADER]: Becoming Leader. State: Replica: 789e1f0ec7034fd9ba76b038312fe76f, State: Running, Role: LEADER
I20260812 09:45:36.953630 22723 consensus_queue.cc:237] T 00000000000000000000000000000000 P 789e1f0ec7034fd9ba76b038312fe76f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "789e1f0ec7034fd9ba76b038312fe76f" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 43059 } }
I20260812 09:45:36.954260 22714 sys_catalog.cc:564] T 00000000000000000000000000000000 P 789e1f0ec7034fd9ba76b038312fe76f [sys.catalog]: configured and running, proceeding with master startup.
I20260812 09:45:36.955401 22728 sys_catalog.cc:455] T 00000000000000000000000000000000 P 789e1f0ec7034fd9ba76b038312fe76f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 789e1f0ec7034fd9ba76b038312fe76f. Latest consensus state: current_term: 1 leader_uuid: "789e1f0ec7034fd9ba76b038312fe76f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "789e1f0ec7034fd9ba76b038312fe76f" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 43059 } } }
I20260812 09:45:36.955546 22728 sys_catalog.cc:458] T 00000000000000000000000000000000 P 789e1f0ec7034fd9ba76b038312fe76f [sys.catalog]: This master's current role is: LEADER
I20260812 09:45:36.955854 22726 sys_catalog.cc:455] T 00000000000000000000000000000000 P 789e1f0ec7034fd9ba76b038312fe76f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "789e1f0ec7034fd9ba76b038312fe76f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "789e1f0ec7034fd9ba76b038312fe76f" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 43059 } } }
I20260812 09:45:36.955952 22738 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 09:45:36.955966 22726 sys_catalog.cc:458] T 00000000000000000000000000000000 P 789e1f0ec7034fd9ba76b038312fe76f [sys.catalog]: This master's current role is: LEADER
I20260812 09:45:36.958798 22738 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 09:45:36.964581 22738 catalog_manager.cc:1347] Generated new cluster ID: ff5ad01ba2ef41719ae3e27c793f233e
I20260812 09:45:36.964656 22738 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 09:45:36.972370 22738 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 09:45:36.973444 22738 catalog_manager.cc:1504] Loading token signing keys...
I20260812 09:45:36.977977 22738 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 789e1f0ec7034fd9ba76b038312fe76f: Generated new TSK 0
I20260812 09:45:36.978734 22738 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 09:45:37.091316 22718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:36.996363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41028 (local address 127.13.99.62:43059)
0812 09:45:36.996522 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:36.996527 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:36.996729 (+ 202us) server_negotiation.cc:407] Connection header received
0812 09:45:36.996905 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:36.996908 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:36.996956 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:36.997045 (+ 89us) server_negotiation.cc:226] Negotiated authn=JWT
0812 09:45:36.997302 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:36.997835 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:36.999074 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:36.999299 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:36.999371 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:36.999707 (+ 336us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0812 09:45:37.090928 (+ 91221us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0812 09:45:37.091194 (+ 266us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41028: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":54}
W20260812 09:45:37.091419 22718 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41028: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260812 09:45:37.091586 22755 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 09:45:36.996295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:43059 (local address 127.0.0.1:41028)
0812 09:45:36.996632 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:36.996647 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:36.996766 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:36.997072 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:36.997075 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:36.997084 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0812 09:45:36.997196 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:36.997202 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:36.997955 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:36.997959 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:36.998951 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:36.998962 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:36.999417 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:36.999421 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:36.999517 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:36.999545 (+ 28us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0812 09:45:37.091182 (+ 91637us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0812 09:45:37.091401 (+ 219us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.13.99.62:43059: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":118,"threads_started":1}
I20260812 09:45:37.092283 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 22618
2026-08-12T09:45:37Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (377 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (129 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (127 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 09:45:37 dist-test-slave-jq4t krb5kdc[22776](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 09:45:37 dist-test-slave-jq4t krb5kdc[22776](info): set up 2 sockets
Aug 12 09:45:37 dist-test-slave-jq4t krb5kdc[22776](info): commencing operation
krb5kdc: starting...
W20260812 09:45:38.626739 13708 mini_kdc.cc:121] Time spent starting KDC: real 1.248s user 0.006s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/krb5kdc/test-user.keytab.
Aug 12 09:45:38 dist-test-slave-jq4t krb5kdc[22776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527938, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T09:45:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T09:45:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.99.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.99.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.99.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab.
Entry for principal kudu/127.13.99.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab.
I20260812 09:45:38.755725 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.62:33633
--webserver_interface=127.13.99.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab
--principal=kudu/127.13.99.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:35901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.99.62:33633
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:45:38.867671 22795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:45:38.867967 22795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:45:38.868042 22795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:45:38.872036 22795 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 09:45:38.872123 22795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:45:38.872164 22795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:45:38.872221 22795 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 09:45:38.872262 22795 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 09:45:38.877377 22795 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:35901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.99.62:33633
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab
--principal=kudu/127.13.99.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.62:33633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.99.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22795
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:45:38.878382 22795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:45:38.879403 22795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 09:45:38.888181 22795 server_base.cc:1034] running on GCE node
W20260812 09:45:38.888145 22805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:38.888163 22801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:38.888181 22800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:45:38.889052 22795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:45:38.890115 22795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:45:38.891306 22795 hybrid_clock.cc:648] HybridClock initialized: now 1786527938891279 us; error 42 us; skew 500 ppm
Aug 12 09:45:38 dist-test-slave-jq4t krb5kdc[22776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527938, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:45:38.898437 22795 init.cc:377] Logged in from keytab as kudu/127.13.99.62@KRBTEST.COM (short username kudu)
I20260812 09:45:38.899380 22795 webserver.cc:460] Webserver started at http://127.13.99.62:35131/ using document root <none> and password file <none>
I20260812 09:45:38.899988 22795 fs_manager.cc:359] Metadata directory not provided
I20260812 09:45:38.900089 22795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:45:38.900326 22795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:45:38.902511 22795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/minicluster-data/master-0/data/instance:
uuid: "f79754c470b7400988bab21a9c55929a"
format_stamp: "Formatted at 2026-08-12 09:45:38 on dist-test-slave-jq4t"
server_key: "89f5c748caa85342e0e075d1888d48ca"
server_key_iv: "4c150e3f31ed942e09966e63febc3d4e"
server_key_version: "encryptionkey@0"
I20260812 09:45:38.903095 22795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/minicluster-data/master-0/wal/instance:
uuid: "f79754c470b7400988bab21a9c55929a"
format_stamp: "Formatted at 2026-08-12 09:45:38 on dist-test-slave-jq4t"
server_key: "89f5c748caa85342e0e075d1888d48ca"
server_key_iv: "4c150e3f31ed942e09966e63febc3d4e"
server_key_version: "encryptionkey@0"
I20260812 09:45:38.907039 22795 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260812 09:45:38.909428 22814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:38.910614 22795 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 09:45:38.910764 22795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/minicluster-data/master-0/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/minicluster-data/master-0/wal
uuid: "f79754c470b7400988bab21a9c55929a"
format_stamp: "Formatted at 2026-08-12 09:45:38 on dist-test-slave-jq4t"
server_key: "89f5c748caa85342e0e075d1888d48ca"
server_key_iv: "4c150e3f31ed942e09966e63febc3d4e"
server_key_version: "encryptionkey@0"
I20260812 09:45:38.910910 22795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:45:38.927486 22795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:45:38.930270 22795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:45:38.930470 22795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:45:38.937523 22795 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.62:33633
I20260812 09:45:38.937552 22894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.62:33633 every 8 connection(s)
I20260812 09:45:38.938455 22795 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
I20260812 09:45:38.941244 22895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:45:38.942039 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 22795
I20260812 09:45:38.942171 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/minicluster-data/master-0/wal/instance
I20260812 09:45:38.942487 13708 external_mini_cluster.cc:1428] Setting key a3dfed62e0827968caca5ffba2a762e0
I20260812 09:45:38.947592 22895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f79754c470b7400988bab21a9c55929a: Bootstrap starting.
I20260812 09:45:38.950064 22895 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f79754c470b7400988bab21a9c55929a: Neither blocks nor log segments found. Creating new log.
I20260812 09:45:38.950840 22895 log.cc:824] T 00000000000000000000000000000000 P f79754c470b7400988bab21a9c55929a: Log is configured to *not* fsync() on all Append() calls
Aug 12 09:45:38 dist-test-slave-jq4t krb5kdc[22776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527938, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
I20260812 09:45:38.953055 22895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f79754c470b7400988bab21a9c55929a: No bootstrap required, opened a new log
I20260812 09:45:38.957007 22901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:38.943984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35196 (local address 127.13.99.62:33633)
0812 09:45:38.944456 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:38.944515 (+ 59us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:38.944550 (+ 35us) server_negotiation.cc:407] Connection header received
0812 09:45:38.945188 (+ 638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:38.945204 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:38.945534 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:38.945880 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:38.946685 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:38.947691 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:38.948591 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:38.949072 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:38.949242 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:38.952214 (+ 2972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:38.952219 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:38.952232 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:38.952258 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:38.954069 (+ 1811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:38.954577 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:38.954582 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:38.954587 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:38.954694 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:38.955028 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:38.955031 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:38.955032 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:38.955419 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:38.955574 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:38.955876 (+ 302us) server_negotiation.cc:299] Negotiation successful
0812 09:45:38.956184 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":122,"threads_started":1}
I20260812 09:45:38.957304 22895 raft_consensus.cc:348] T 00000000000000000000000000000000 P f79754c470b7400988bab21a9c55929a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f79754c470b7400988bab21a9c55929a" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 33633 } }
I20260812 09:45:38.957561 22895 raft_consensus.cc:374] T 00000000000000000000000000000000 P f79754c470b7400988bab21a9c55929a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:45:38.957640 22895 raft_consensus.cc:729] T 00000000000000000000000000000000 P f79754c470b7400988bab21a9c55929a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f79754c470b7400988bab21a9c55929a, State: Initialized, Role: FOLLOWER
I20260812 09:45:38.958192 22895 consensus_queue.cc:260] T 00000000000000000000000000000000 P f79754c470b7400988bab21a9c55929a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f79754c470b7400988bab21a9c55929a" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 33633 } }
I20260812 09:45:38.958362 22895 raft_consensus.cc:388] T 00000000000000000000000000000000 P f79754c470b7400988bab21a9c55929a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 09:45:38.958431 22895 raft_consensus.cc:482] T 00000000000000000000000000000000 P f79754c470b7400988bab21a9c55929a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 09:45:38.958527 22895 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f79754c470b7400988bab21a9c55929a [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:45:38.959591 22895 raft_consensus.cc:504] T 00000000000000000000000000000000 P f79754c470b7400988bab21a9c55929a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f79754c470b7400988bab21a9c55929a" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 33633 } }
I20260812 09:45:38.959892 22895 leader_election.cc:302] T 00000000000000000000000000000000 P f79754c470b7400988bab21a9c55929a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f79754c470b7400988bab21a9c55929a; no voters:
I20260812 09:45:38.960142 22895 leader_election.cc:288] T 00000000000000000000000000000000 P f79754c470b7400988bab21a9c55929a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 09:45:38.960294 22905 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f79754c470b7400988bab21a9c55929a [term 1 FOLLOWER]: Leader election won for term 1
I20260812 09:45:38.960541 22905 raft_consensus.cc:686] T 00000000000000000000000000000000 P f79754c470b7400988bab21a9c55929a [term 1 LEADER]: Becoming Leader. State: Replica: f79754c470b7400988bab21a9c55929a, State: Running, Role: LEADER
I20260812 09:45:38.960876 22905 consensus_queue.cc:237] T 00000000000000000000000000000000 P f79754c470b7400988bab21a9c55929a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f79754c470b7400988bab21a9c55929a" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 33633 } }
I20260812 09:45:38.961436 22895 sys_catalog.cc:564] T 00000000000000000000000000000000 P f79754c470b7400988bab21a9c55929a [sys.catalog]: configured and running, proceeding with master startup.
I20260812 09:45:38.962657 22906 sys_catalog.cc:455] T 00000000000000000000000000000000 P f79754c470b7400988bab21a9c55929a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f79754c470b7400988bab21a9c55929a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f79754c470b7400988bab21a9c55929a" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 33633 } } }
I20260812 09:45:38.962690 22908 sys_catalog.cc:455] T 00000000000000000000000000000000 P f79754c470b7400988bab21a9c55929a [sys.catalog]: SysCatalogTable state changed. Reason: New leader f79754c470b7400988bab21a9c55929a. Latest consensus state: current_term: 1 leader_uuid: "f79754c470b7400988bab21a9c55929a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f79754c470b7400988bab21a9c55929a" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 33633 } } }
I20260812 09:45:38.962862 22906 sys_catalog.cc:458] T 00000000000000000000000000000000 P f79754c470b7400988bab21a9c55929a [sys.catalog]: This master's current role is: LEADER
I20260812 09:45:38.962867 22908 sys_catalog.cc:458] T 00000000000000000000000000000000 P f79754c470b7400988bab21a9c55929a [sys.catalog]: This master's current role is: LEADER
I20260812 09:45:38.963754 22917 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 09:45:38.967150 22917 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 09:45:38.972725 22917 catalog_manager.cc:1347] Generated new cluster ID: c64002b9306c4b739058b39ef54c694d
I20260812 09:45:38.972841 22917 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 09:45:38.996743 22917 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 09:45:38.997733 22917 catalog_manager.cc:1504] Loading token signing keys...
I20260812 09:45:39.009316 22917 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f79754c470b7400988bab21a9c55929a: Generated new TSK 0
I20260812 09:45:39.010023 22917 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.99.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.99.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.99.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.1.keytab.
Entry for principal kudu/127.13.99.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.1.keytab.
I20260812 09:45:39.045293 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.99.1:0
--local_ip_for_outbound_sockets=127.13.99.1
--webserver_interface=127.13.99.1
--webserver_port=0
--tserver_master_addrs=127.13.99.62:33633
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.1.keytab
--principal=kudu/127.13.99.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:35901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:45:39.173188 22934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:45:39.173446 22934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:45:39.173488 22934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:45:39.177423 22934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:45:39.177471 22934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:45:39.177528 22934 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.99.1
I20260812 09:45:39.183004 22934 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:35901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.1.keytab
--principal=kudu/127.13.99.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.99.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.99.1
--webserver_port=0
--tserver_master_addrs=127.13.99.62:33633
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22934
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.99.1
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:45:39.184101 22934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:45:39.185133 22934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:45:39.194828 22949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:39.194860 22944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:39.194905 22943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:45:39.194998 22934 server_base.cc:1034] running on GCE node
I20260812 09:45:39.195581 22934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:45:39.196185 22934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:45:39.197395 22934 hybrid_clock.cc:648] HybridClock initialized: now 1786527939197353 us; error 53 us; skew 500 ppm
Aug 12 09:45:39 dist-test-slave-jq4t krb5kdc[22776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527939, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:45:39.200191 22934 init.cc:377] Logged in from keytab as kudu/127.13.99.1@KRBTEST.COM (short username kudu)
I20260812 09:45:39.201109 22934 webserver.cc:460] Webserver started at http://127.13.99.1:45399/ using document root <none> and password file <none>
I20260812 09:45:39.201681 22934 fs_manager.cc:359] Metadata directory not provided
I20260812 09:45:39.201776 22934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:45:39.202064 22934 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:45:39.204134 22934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/minicluster-data/ts-0/data/instance:
uuid: "da0d2d5ff4ef4aad8263bd2ea85a5511"
format_stamp: "Formatted at 2026-08-12 09:45:39 on dist-test-slave-jq4t"
server_key: "5d8646e670f654eb5a0b6798dcbc2fea"
server_key_iv: "e6afc5ed560b45a7980399ceb4da810e"
server_key_version: "encryptionkey@0"
I20260812 09:45:39.204747 22934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/minicluster-data/ts-0/wal/instance:
uuid: "da0d2d5ff4ef4aad8263bd2ea85a5511"
format_stamp: "Formatted at 2026-08-12 09:45:39 on dist-test-slave-jq4t"
server_key: "5d8646e670f654eb5a0b6798dcbc2fea"
server_key_iv: "e6afc5ed560b45a7980399ceb4da810e"
server_key_version: "encryptionkey@0"
I20260812 09:45:39.208595 22934 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 09:45:39.211098 22958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:39.212428 22934 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 09:45:39.212613 22934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/minicluster-data/ts-0/wal
uuid: "da0d2d5ff4ef4aad8263bd2ea85a5511"
format_stamp: "Formatted at 2026-08-12 09:45:39 on dist-test-slave-jq4t"
server_key: "5d8646e670f654eb5a0b6798dcbc2fea"
server_key_iv: "e6afc5ed560b45a7980399ceb4da810e"
server_key_version: "encryptionkey@0"
I20260812 09:45:39.212735 22934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:45:39.222337 22934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:45:39.225188 22934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:45:39.225385 22934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:45:39.225963 22934 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 09:45:39.226841 22934 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:45:39.226912 22934 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:39.226986 22934 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:45:39.227036 22934 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:39.236562 22934 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.1:38371
I20260812 09:45:39.236603 23115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.1:38371 every 8 connection(s)
I20260812 09:45:39.237530 22934 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/minicluster-data/ts-0/data/info.pb
I20260812 09:45:39.242828 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 22934
I20260812 09:45:39.242942 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/minicluster-data/ts-0/wal/instance
I20260812 09:45:39.243255 13708 external_mini_cluster.cc:1428] Setting key 77ac6ccc5adc7ec170214db2f69605c0
Aug 12 09:45:39 dist-test-slave-jq4t krb5kdc[22776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527939, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.1@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
I20260812 09:45:39.253178 22901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:39.239522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.1:55559 (local address 127.13.99.62:33633)
0812 09:45:39.239670 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:39.239673 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:39.240605 (+ 932us) server_negotiation.cc:407] Connection header received
0812 09:45:39.242066 (+ 1461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:39.242070 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:39.242134 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:39.242242 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:39.243613 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:39.244269 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:39.245569 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:39.245808 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:39.245933 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:39.248831 (+ 2898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:39.248838 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:39.248844 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:39.248886 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:39.250671 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:39.251223 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:39.251230 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:39.251235 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:39.251341 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:39.251698 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:39.251704 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:39.251708 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:39.252031 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:39.252156 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:39.252776 (+ 620us) server_negotiation.cc:299] Negotiation successful
0812 09:45:39.252949 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260812 09:45:39.253798 23120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:39.239838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:33633 (local address 127.13.99.1:55559)
0812 09:45:39.240328 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:39.240381 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:39.241833 (+ 1452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:39.242434 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:39.242441 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:39.242818 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:45:39.243450 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:39.243467 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:39.244410 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:39.244414 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:39.245431 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:39.245442 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:39.245975 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:39.245980 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:39.246158 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:39.246851 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:45:39.246867 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:45:39.248657 (+ 1790us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:45:39.250820 (+ 2163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:39.250825 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:39.250837 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:39.251096 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:39.251447 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:39.251449 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:39.251451 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:39.251581 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:39.252169 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:45:39.252176 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:45:39.252449 (+ 273us) client_negotiation.cc:769] Sending connection context
0812 09:45:39.252665 (+ 216us) client_negotiation.cc:240] Negotiation successful
0812 09:45:39.252920 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":120,"threads_started":1}
WARNING: no policy specified for kudu/127.13.99.2@KRBTEST.COM; defaulting to no policy
I20260812 09:45:39.255059 23118 heartbeater.cc:347] Connected to a master server at 127.13.99.62:33633
I20260812 09:45:39.255271 23118 heartbeater.cc:464] Registering TS with master...
I20260812 09:45:39.255805 23118 heartbeater.cc:511] Master 127.13.99.62:33633 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.99.2@KRBTEST.COM" created.
I20260812 09:45:39.257380 22840 ts_manager.cc:194] Registered new tserver with Master: da0d2d5ff4ef4aad8263bd2ea85a5511 (127.13.99.1:38371)
I20260812 09:45:39.258519 22840 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.99.1@KRBTEST.COM'} at 127.13.99.1:55559
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.99.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.2.keytab.
Entry for principal kudu/127.13.99.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.2.keytab.
I20260812 09:45:39.276049 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.99.2:0
--local_ip_for_outbound_sockets=127.13.99.2
--webserver_interface=127.13.99.2
--webserver_port=0
--tserver_master_addrs=127.13.99.62:33633
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.2.keytab
--principal=kudu/127.13.99.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:35901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:45:39.394820 23124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:45:39.395074 23124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:45:39.395118 23124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:45:39.399196 23124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:45:39.399263 23124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:45:39.399322 23124 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.99.2
I20260812 09:45:39.404747 23124 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:35901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.2.keytab
--principal=kudu/127.13.99.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.99.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.99.2
--webserver_port=0
--tserver_master_addrs=127.13.99.62:33633
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23124
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.99.2
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:45:39.405838 23124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:45:39.406937 23124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:45:39.416946 23134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:39.416968 23135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:39.416965 23138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:45:39.417394 23124 server_base.cc:1034] running on GCE node
I20260812 09:45:39.417730 23124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:45:39.418286 23124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:45:39.419448 23124 hybrid_clock.cc:648] HybridClock initialized: now 1786527939419415 us; error 45 us; skew 500 ppm
Aug 12 09:45:39 dist-test-slave-jq4t krb5kdc[22776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527939, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:45:39.422195 23124 init.cc:377] Logged in from keytab as kudu/127.13.99.2@KRBTEST.COM (short username kudu)
I20260812 09:45:39.423115 23124 webserver.cc:460] Webserver started at http://127.13.99.2:39579/ using document root <none> and password file <none>
I20260812 09:45:39.423645 23124 fs_manager.cc:359] Metadata directory not provided
I20260812 09:45:39.423703 23124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:45:39.423866 23124 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:45:39.425951 23124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/minicluster-data/ts-1/data/instance:
uuid: "fca7b46f950545eb9a3496d6b8449d73"
format_stamp: "Formatted at 2026-08-12 09:45:39 on dist-test-slave-jq4t"
server_key: "ea7fade032e3f650c06e4e7ec8696903"
server_key_iv: "bf000bbdd10c244c1ebebe7aba640ce0"
server_key_version: "encryptionkey@0"
I20260812 09:45:39.426488 23124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/minicluster-data/ts-1/wal/instance:
uuid: "fca7b46f950545eb9a3496d6b8449d73"
format_stamp: "Formatted at 2026-08-12 09:45:39 on dist-test-slave-jq4t"
server_key: "ea7fade032e3f650c06e4e7ec8696903"
server_key_iv: "bf000bbdd10c244c1ebebe7aba640ce0"
server_key_version: "encryptionkey@0"
I20260812 09:45:39.430298 23124 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 09:45:39.433277 23147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:39.434640 23124 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 09:45:39.434902 23124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/minicluster-data/ts-1/wal
uuid: "fca7b46f950545eb9a3496d6b8449d73"
format_stamp: "Formatted at 2026-08-12 09:45:39 on dist-test-slave-jq4t"
server_key: "ea7fade032e3f650c06e4e7ec8696903"
server_key_iv: "bf000bbdd10c244c1ebebe7aba640ce0"
server_key_version: "encryptionkey@0"
I20260812 09:45:39.435034 23124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:45:39.446565 23124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:45:39.450026 23124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:45:39.450233 23124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:45:39.450856 23124 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 09:45:39.451725 23124 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:45:39.451797 23124 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:39.451866 23124 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:45:39.451927 23124 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:39.461056 23124 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.2:44641
I20260812 09:45:39.461086 23310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.2:44641 every 8 connection(s)
I20260812 09:45:39.462002 23124 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/minicluster-data/ts-1/data/info.pb
I20260812 09:45:39.463349 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 23124
I20260812 09:45:39.463467 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/minicluster-data/ts-1/wal/instance
I20260812 09:45:39.463711 13708 external_mini_cluster.cc:1428] Setting key c05587ca18c9dc7aea446454e2434329
Aug 12 09:45:39 dist-test-slave-jq4t krb5kdc[22776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527939, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.2@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.99.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.99.3@KRBTEST.COM" created.
I20260812 09:45:39.477809 22901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:39.464704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.2:41065 (local address 127.13.99.62:33633)
0812 09:45:39.464833 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:39.464838 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:39.465231 (+ 393us) server_negotiation.cc:407] Connection header received
0812 09:45:39.466655 (+ 1424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:39.466662 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:39.466726 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:39.466821 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:39.468372 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:39.468971 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:39.470828 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:39.471053 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:39.471131 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:39.473817 (+ 2686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:39.473821 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:39.473826 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:39.473858 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:39.475508 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:39.476085 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:39.476094 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:39.476095 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:39.476174 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:39.476563 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:39.476566 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:39.476568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:39.476762 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:39.476895 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:39.477515 (+ 620us) server_negotiation.cc:299] Negotiation successful
0812 09:45:39.477644 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260812 09:45:39.478693 23313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:39.464368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:33633 (local address 127.13.99.2:41065)
0812 09:45:39.465086 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:39.465137 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:39.466439 (+ 1302us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:39.466974 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:39.466981 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:39.467507 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:45:39.468207 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:39.468227 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:39.469105 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:39.469112 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:39.470674 (+ 1562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:39.470690 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:39.471212 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:39.471218 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:39.471429 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:39.472087 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:45:39.472103 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:45:39.473645 (+ 1542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:45:39.475635 (+ 1990us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:39.475641 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:39.475657 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:39.475965 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:39.476268 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:39.476271 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:39.476274 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:39.476424 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:39.476924 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:45:39.476932 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:45:39.477200 (+ 268us) client_negotiation.cc:769] Sending connection context
0812 09:45:39.477478 (+ 278us) client_negotiation.cc:240] Negotiation successful
0812 09:45:39.477806 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":177,"threads_started":1}
I20260812 09:45:39.480684 23311 heartbeater.cc:347] Connected to a master server at 127.13.99.62:33633
I20260812 09:45:39.480970 23311 heartbeater.cc:464] Registering TS with master...
I20260812 09:45:39.481591 23311 heartbeater.cc:511] Master 127.13.99.62:33633 requested a full tablet report, sending...
I20260812 09:45:39.482805 22840 ts_manager.cc:194] Registered new tserver with Master: fca7b46f950545eb9a3496d6b8449d73 (127.13.99.2:44641)
I20260812 09:45:39.483410 22840 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.99.2@KRBTEST.COM'} at 127.13.99.2:41065
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.99.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.3.keytab.
Entry for principal kudu/127.13.99.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.3.keytab.
I20260812 09:45:39.494699 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.99.3:0
--local_ip_for_outbound_sockets=127.13.99.3
--webserver_interface=127.13.99.3
--webserver_port=0
--tserver_master_addrs=127.13.99.62:33633
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.3.keytab
--principal=kudu/127.13.99.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:35901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:45:39.609503 23317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:45:39.609817 23317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:45:39.609896 23317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:45:39.614100 23317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:45:39.614207 23317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:45:39.614280 23317 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.99.3
I20260812 09:45:39.619647 23317 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:35901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.3.keytab
--principal=kudu/127.13.99.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.99.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.99.3
--webserver_port=0
--tserver_master_addrs=127.13.99.62:33633
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23317
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.99.3
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:45:39.620711 23317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:45:39.621675 23317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:45:39.631237 23329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:39.631326 23327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:45:39.631368 23317 server_base.cc:1034] running on GCE node
W20260812 09:45:39.631237 23323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:45:39.631974 23317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:45:39.632611 23317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:45:39.633771 23317 hybrid_clock.cc:648] HybridClock initialized: now 1786527939633741 us; error 43 us; skew 500 ppm
Aug 12 09:45:39 dist-test-slave-jq4t krb5kdc[22776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527939, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:45:39.636588 23317 init.cc:377] Logged in from keytab as kudu/127.13.99.3@KRBTEST.COM (short username kudu)
I20260812 09:45:39.637452 23317 webserver.cc:460] Webserver started at http://127.13.99.3:38467/ using document root <none> and password file <none>
I20260812 09:45:39.638010 23317 fs_manager.cc:359] Metadata directory not provided
I20260812 09:45:39.638072 23317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:45:39.638337 23317 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:45:39.640466 23317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/minicluster-data/ts-2/data/instance:
uuid: "e2b8389021084d0b847a21578962e094"
format_stamp: "Formatted at 2026-08-12 09:45:39 on dist-test-slave-jq4t"
server_key: "8a888f946d58b18b38ee6549e3ad8320"
server_key_iv: "dc6791c5babf7e874b09c82d1398e103"
server_key_version: "encryptionkey@0"
I20260812 09:45:39.641116 23317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/minicluster-data/ts-2/wal/instance:
uuid: "e2b8389021084d0b847a21578962e094"
format_stamp: "Formatted at 2026-08-12 09:45:39 on dist-test-slave-jq4t"
server_key: "8a888f946d58b18b38ee6549e3ad8320"
server_key_iv: "dc6791c5babf7e874b09c82d1398e103"
server_key_version: "encryptionkey@0"
I20260812 09:45:39.644939 23317 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 09:45:39.647459 23340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:39.648844 23317 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 09:45:39.648970 23317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/minicluster-data/ts-2/wal
uuid: "e2b8389021084d0b847a21578962e094"
format_stamp: "Formatted at 2026-08-12 09:45:39 on dist-test-slave-jq4t"
server_key: "8a888f946d58b18b38ee6549e3ad8320"
server_key_iv: "dc6791c5babf7e874b09c82d1398e103"
server_key_version: "encryptionkey@0"
I20260812 09:45:39.649088 23317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:45:39.663465 23317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:45:39.666518 23317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:45:39.666734 23317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:45:39.667366 23317 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 09:45:39.668233 23317 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:45:39.668283 23317 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:39.668352 23317 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:45:39.668388 23317 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:39.677454 23317 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.3:38969
I20260812 09:45:39.677472 23513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.3:38969 every 8 connection(s)
I20260812 09:45:39.678372 23317 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/minicluster-data/ts-2/data/info.pb
I20260812 09:45:39.681646 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 23317
I20260812 09:45:39.681754 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/minicluster-data/ts-2/wal/instance
I20260812 09:45:39.682058 13708 external_mini_cluster.cc:1428] Setting key a0a2a5be47729ba112c44f63c987a90a
Aug 12 09:45:39 dist-test-slave-jq4t krb5kdc[22776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527939, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.3@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
I20260812 09:45:39.692049 22901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:39.680232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.3:46459 (local address 127.13.99.62:33633)
0812 09:45:39.680381 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:39.680385 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:39.681290 (+ 905us) server_negotiation.cc:407] Connection header received
0812 09:45:39.682714 (+ 1424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:39.682718 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:39.682777 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:39.682884 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:39.684342 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:39.684992 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:39.686184 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:39.686403 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:39.686497 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:39.689069 (+ 2572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:39.689072 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:39.689074 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:39.689097 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:39.690305 (+ 1208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:39.690755 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:39.690758 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:39.690759 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:39.690843 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:39.691145 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:39.691147 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:39.691148 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:39.691311 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:39.691387 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:39.691832 (+ 445us) server_negotiation.cc:299] Negotiation successful
0812 09:45:39.691932 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260812 09:45:39.692902 23517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:39.680677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:33633 (local address 127.13.99.3:46459)
0812 09:45:39.681135 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:39.681178 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:39.682501 (+ 1323us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:39.683080 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:39.683087 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:39.683624 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:45:39.684138 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:39.684152 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:39.685160 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:39.685166 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:39.686065 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:39.686075 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:39.686568 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:39.686572 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:39.686722 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:39.687384 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:45:39.687404 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:45:39.688916 (+ 1512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:45:39.690412 (+ 1496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:39.690417 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:39.690428 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:39.690667 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:39.690935 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:39.690937 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:39.690938 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:39.691064 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:39.691401 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:45:39.691406 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:45:39.691625 (+ 219us) client_negotiation.cc:769] Sending connection context
0812 09:45:39.691779 (+ 154us) client_negotiation.cc:240] Negotiation successful
0812 09:45:39.692043 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":117,"threads_started":1}
I20260812 09:45:39.694085 23514 heartbeater.cc:347] Connected to a master server at 127.13.99.62:33633
I20260812 09:45:39.694324 23514 heartbeater.cc:464] Registering TS with master...
I20260812 09:45:39.694808 23514 heartbeater.cc:511] Master 127.13.99.62:33633 requested a full tablet report, sending...
I20260812 09:45:39.695871 22840 ts_manager.cc:194] Registered new tserver with Master: e2b8389021084d0b847a21578962e094 (127.13.99.3:38969)
I20260812 09:45:39.696439 13708 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 09:45:39.696571 22840 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.99.3@KRBTEST.COM'} at 127.13.99.3:46459
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/krb5kdc/test-admin.keytab.
Aug 12 09:45:39 dist-test-slave-jq4t krb5kdc[22776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:45:39.713788 13708 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260812 09:45:39.719309 23523 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 09:45:39.715561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58812 (local address 127.13.99.1:38371)
0812 09:45:39.718025 (+ 2464us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:39.718044 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:39.718077 (+ 33us) server_negotiation.cc:407] Connection header received
0812 09:45:39.718132 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:39.718137 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:39.718328 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:39.718452 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:39.719097 (+ 645us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58812: BlockingRecv error: recv got EOF from 127.0.0.1:58812 (error 108)
Metrics: {"server-negotiator.queue_time_us":2359,"thread_start_us":2283,"threads_started":1}
W20260812 09:45:39.719444 13708 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/krb5kdc/krb5cc' of type FILE
Aug 12 09:45:39 dist-test-slave-jq4t krb5kdc[22776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:45:39.720605 13708 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 12 09:45:39 dist-test-slave-jq4t krb5kdc[22776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:45:39.721585 13708 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 12 09:45:39 dist-test-slave-jq4t krb5kdc[22776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.99.1@KRBTEST.COM
I20260812 09:45:39.731197 23523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:39.722445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58822 (local address 127.13.99.1:38371)
0812 09:45:39.722608 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:39.722611 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:39.722679 (+ 68us) server_negotiation.cc:407] Connection header received
0812 09:45:39.722766 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:39.722769 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:39.722807 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:39.722876 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:39.723356 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:39.724130 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:39.725132 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:39.725567 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:39.725681 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:39.727946 (+ 2265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:39.727955 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:39.727968 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:39.727991 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:39.729574 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:39.729970 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:39.729975 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:39.729976 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:39.730056 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:39.730284 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:39.730286 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:39.730287 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:39.730515 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:39.730676 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:39.730868 (+ 192us) server_negotiation.cc:299] Negotiation successful
0812 09:45:39.731061 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260812 09:45:39.734902 23523 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 09:45:39.734029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58830 (local address 127.13.99.1:38371)
0812 09:45:39.734154 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:39.734158 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:39.734267 (+ 109us) server_negotiation.cc:407] Connection header received
0812 09:45:39.734338 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:39.734340 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:39.734375 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:39.734436 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:39.734798 (+ 362us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58830: BlockingRecv error: recv got EOF from 127.0.0.1:58830 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260812 09:45:39.735157 13708 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/krb5kdc/krb5cc)
Aug 12 09:45:39 dist-test-slave-jq4t krb5kdc[22776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:45:39.736068 13708 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 12 09:45:39 dist-test-slave-jq4t krb5kdc[22776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:45:39.736986 13708 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 12 09:45:39 dist-test-slave-jq4t krb5kdc[22776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.99.1@KRBTEST.COM
I20260812 09:45:39.746296 23523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:39.738236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58836 (local address 127.13.99.1:38371)
0812 09:45:39.738409 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:39.738413 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:39.738865 (+ 452us) server_negotiation.cc:407] Connection header received
0812 09:45:39.739002 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:39.739005 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:39.739053 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:39.739133 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:39.739762 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:39.740519 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:39.741297 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:39.741543 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:39.741653 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:39.743699 (+ 2046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:39.743702 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:39.743704 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:39.743725 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:39.744980 (+ 1255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:39.745349 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:39.745351 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:39.745352 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:39.745423 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:39.745648 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:39.745650 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:39.745651 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:39.745796 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:39.745885 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:39.746106 (+ 221us) server_negotiation.cc:299] Negotiation successful
0812 09:45:39.746196 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260812 09:45:39.749068 23523 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 09:45:39.747904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58846 (local address 127.13.99.1:38371)
0812 09:45:39.748023 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:39.748039 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:39.748262 (+ 223us) server_negotiation.cc:407] Connection header received
0812 09:45:39.748308 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:39.748313 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:39.748364 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:39.748460 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:39.748980 (+ 520us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58846: BlockingRecv error: recv got EOF from 127.0.0.1:58846 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260812 09:45:39.749193 13708 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786527905902985-13708-0/krb5kdc/krb5cc' of type FILE
Aug 12 09:45:39 dist-test-slave-jq4t krb5kdc[22776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:45:39.750183 13708 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 12 09:45:39 dist-test-slave-jq4t krb5kdc[22776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:45:39.751137 13708 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 12 09:45:39 dist-test-slave-jq4t krb5kdc[22776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.99.1@KRBTEST.COM
I20260812 09:45:39.759042 23523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:39.751901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58856 (local address 127.13.99.1:38371)
0812 09:45:39.752016 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:39.752020 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:39.752124 (+ 104us) server_negotiation.cc:407] Connection header received
0812 09:45:39.752242 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:39.752245 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:39.752284 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:39.752352 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:39.752859 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:39.753508 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:39.754240 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:39.754466 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:39.754546 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:39.756575 (+ 2029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:39.756578 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:39.756579 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:39.756600 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:39.757762 (+ 1162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:39.758138 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:39.758140 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:39.758142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:39.758211 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:39.758432 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:39.758434 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:39.758435 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:39.758573 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:39.758660 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:39.758846 (+ 186us) server_negotiation.cc:299] Negotiation successful
0812 09:45:39.758947 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260812 09:45:39.760082 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 22934
I20260812 09:45:39.778967 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 23124
I20260812 09:45:39.795027 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 23317
I20260812 09:45:39.808226 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 22795
2026-08-12T09:45:39Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2456 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 09:45:39 dist-test-slave-jq4t krb5kdc[23544](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 09:45:39 dist-test-slave-jq4t krb5kdc[23544](info): set up 2 sockets
Aug 12 09:45:39 dist-test-slave-jq4t krb5kdc[23544](info): commencing operation
krb5kdc: starting...
W20260812 09:45:41.082515 13708 mini_kdc.cc:121] Time spent starting KDC: real 1.248s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/krb5kdc/test-user.keytab.
Aug 12 09:45:41 dist-test-slave-jq4t krb5kdc[23544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527941, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T09:45:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T09:45:41Z Disabled control of system clock
WARNING: no policy specified for oryx/127.13.99.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.99.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.99.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/krb5kdc/oryx_127.13.99.62.keytab.
Entry for principal oryx/127.13.99.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/krb5kdc/oryx_127.13.99.62.keytab.
I20260812 09:45:41.208458 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.62:43093
--webserver_interface=127.13.99.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/krb5kdc/oryx_127.13.99.62.keytab
--principal=oryx/127.13.99.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:45813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.99.62:43093
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:45:41.327572 23562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:45:41.327816 23562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:45:41.327854 23562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:45:41.331790 23562 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 09:45:41.331848 23562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:45:41.331868 23562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:45:41.331884 23562 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 09:45:41.331900 23562 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 09:45:41.337018 23562 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:45813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.99.62:43093
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/krb5kdc/oryx_127.13.99.62.keytab
--principal=oryx/127.13.99.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.62:43093
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.99.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23562
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:45:41.338096 23562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:45:41.339056 23562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:45:41.347700 23572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:41.347708 23574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:41.347708 23571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:45:41.347980 23562 server_base.cc:1034] running on GCE node
I20260812 09:45:41.348737 23562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:45:41.349846 23562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:45:41.351044 23562 hybrid_clock.cc:648] HybridClock initialized: now 1786527941351011 us; error 49 us; skew 500 ppm
Aug 12 09:45:41 dist-test-slave-jq4t krb5kdc[23544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527941, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.99.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:45:41.354187 23562 init.cc:377] Logged in from keytab as oryx/127.13.99.62@KRBTEST.COM (short username oryx)
I20260812 09:45:41.355145 23562 webserver.cc:460] Webserver started at http://127.13.99.62:40085/ using document root <none> and password file <none>
I20260812 09:45:41.355724 23562 fs_manager.cc:359] Metadata directory not provided
I20260812 09:45:41.355816 23562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:45:41.356062 23562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:45:41.358299 23562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/master-0/data/instance:
uuid: "f8aaf46e4a6d4531ab509eb45d22e2cd"
format_stamp: "Formatted at 2026-08-12 09:45:41 on dist-test-slave-jq4t"
server_key: "672f0f852e05e81da80f4beaa54e9efb"
server_key_iv: "e02b606a6010848866c3b8036f26e8f4"
server_key_version: "encryptionkey@0"
I20260812 09:45:41.358868 23562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/master-0/wal/instance:
uuid: "f8aaf46e4a6d4531ab509eb45d22e2cd"
format_stamp: "Formatted at 2026-08-12 09:45:41 on dist-test-slave-jq4t"
server_key: "672f0f852e05e81da80f4beaa54e9efb"
server_key_iv: "e02b606a6010848866c3b8036f26e8f4"
server_key_version: "encryptionkey@0"
I20260812 09:45:41.362690 23562 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260812 09:45:41.365231 23584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:41.366439 23562 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260812 09:45:41.366583 23562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/master-0/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/master-0/wal
uuid: "f8aaf46e4a6d4531ab509eb45d22e2cd"
format_stamp: "Formatted at 2026-08-12 09:45:41 on dist-test-slave-jq4t"
server_key: "672f0f852e05e81da80f4beaa54e9efb"
server_key_iv: "e02b606a6010848866c3b8036f26e8f4"
server_key_version: "encryptionkey@0"
I20260812 09:45:41.366696 23562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:45:41.386942 23562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:45:41.390484 23562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:45:41.390699 23562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:45:41.398211 23562 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.62:43093
I20260812 09:45:41.398541 23654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.62:43093 every 8 connection(s)
I20260812 09:45:41.399331 23562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
I20260812 09:45:41.402185 23655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:45:41.407775 23655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8aaf46e4a6d4531ab509eb45d22e2cd: Bootstrap starting.
I20260812 09:45:41.408776 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 23562
I20260812 09:45:41.408921 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/master-0/wal/instance
I20260812 09:45:41.409255 13708 external_mini_cluster.cc:1428] Setting key 4d0525af042fc237822561c08f64b4d1
I20260812 09:45:41.410751 23655 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f8aaf46e4a6d4531ab509eb45d22e2cd: Neither blocks nor log segments found. Creating new log.
I20260812 09:45:41.411497 23655 log.cc:824] T 00000000000000000000000000000000 P f8aaf46e4a6d4531ab509eb45d22e2cd: Log is configured to *not* fsync() on all Append() calls
I20260812 09:45:41.413486 23655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8aaf46e4a6d4531ab509eb45d22e2cd: No bootstrap required, opened a new log
I20260812 09:45:41.416853 23655 raft_consensus.cc:348] T 00000000000000000000000000000000 P f8aaf46e4a6d4531ab509eb45d22e2cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8aaf46e4a6d4531ab509eb45d22e2cd" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 43093 } }
I20260812 09:45:41.417098 23655 raft_consensus.cc:374] T 00000000000000000000000000000000 P f8aaf46e4a6d4531ab509eb45d22e2cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:45:41.417183 23655 raft_consensus.cc:729] T 00000000000000000000000000000000 P f8aaf46e4a6d4531ab509eb45d22e2cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8aaf46e4a6d4531ab509eb45d22e2cd, State: Initialized, Role: FOLLOWER
I20260812 09:45:41.417650 23655 consensus_queue.cc:260] T 00000000000000000000000000000000 P f8aaf46e4a6d4531ab509eb45d22e2cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8aaf46e4a6d4531ab509eb45d22e2cd" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 43093 } }
I20260812 09:45:41.417805 23655 raft_consensus.cc:388] T 00000000000000000000000000000000 P f8aaf46e4a6d4531ab509eb45d22e2cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 09:45:41.417903 23655 raft_consensus.cc:482] T 00000000000000000000000000000000 P f8aaf46e4a6d4531ab509eb45d22e2cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 09:45:41.418020 23655 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f8aaf46e4a6d4531ab509eb45d22e2cd [term 0 FOLLOWER]: Advancing to term 1
Aug 12 09:45:41 dist-test-slave-jq4t krb5kdc[23544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527941, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM
I20260812 09:45:41.419076 23655 raft_consensus.cc:504] T 00000000000000000000000000000000 P f8aaf46e4a6d4531ab509eb45d22e2cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8aaf46e4a6d4531ab509eb45d22e2cd" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 43093 } }
I20260812 09:45:41.419426 23655 leader_election.cc:302] T 00000000000000000000000000000000 P f8aaf46e4a6d4531ab509eb45d22e2cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f8aaf46e4a6d4531ab509eb45d22e2cd; no voters:
I20260812 09:45:41.419773 23655 leader_election.cc:288] T 00000000000000000000000000000000 P f8aaf46e4a6d4531ab509eb45d22e2cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 09:45:41.420033 23661 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f8aaf46e4a6d4531ab509eb45d22e2cd [term 1 FOLLOWER]: Leader election won for term 1
I20260812 09:45:41.420307 23661 raft_consensus.cc:686] T 00000000000000000000000000000000 P f8aaf46e4a6d4531ab509eb45d22e2cd [term 1 LEADER]: Becoming Leader. State: Replica: f8aaf46e4a6d4531ab509eb45d22e2cd, State: Running, Role: LEADER
I20260812 09:45:41.420642 23661 consensus_queue.cc:237] T 00000000000000000000000000000000 P f8aaf46e4a6d4531ab509eb45d22e2cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8aaf46e4a6d4531ab509eb45d22e2cd" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 43093 } }
I20260812 09:45:41.421080 23655 sys_catalog.cc:564] T 00000000000000000000000000000000 P f8aaf46e4a6d4531ab509eb45d22e2cd [sys.catalog]: configured and running, proceeding with master startup.
I20260812 09:45:41.421875 23663 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8aaf46e4a6d4531ab509eb45d22e2cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8aaf46e4a6d4531ab509eb45d22e2cd. Latest consensus state: current_term: 1 leader_uuid: "f8aaf46e4a6d4531ab509eb45d22e2cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8aaf46e4a6d4531ab509eb45d22e2cd" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 43093 } } }
I20260812 09:45:41.422009 23663 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8aaf46e4a6d4531ab509eb45d22e2cd [sys.catalog]: This master's current role is: LEADER
I20260812 09:45:41.422377 23662 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8aaf46e4a6d4531ab509eb45d22e2cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f8aaf46e4a6d4531ab509eb45d22e2cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8aaf46e4a6d4531ab509eb45d22e2cd" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 43093 } } }
I20260812 09:45:41.422468 23662 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8aaf46e4a6d4531ab509eb45d22e2cd [sys.catalog]: This master's current role is: LEADER
I20260812 09:45:41.424870 23658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:41.410444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54464 (local address 127.13.99.62:43093)
0812 09:45:41.410948 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:41.410964 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:41.410999 (+ 35us) server_negotiation.cc:407] Connection header received
0812 09:45:41.411723 (+ 724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:41.411743 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:41.412149 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:41.412563 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:41.413371 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:41.414407 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:41.415257 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:41.415738 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:41.415912 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:41.418735 (+ 2823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:41.418739 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:41.418752 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:41.418779 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:41.420932 (+ 2153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:41.421517 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:41.421523 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:41.421529 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:41.421664 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:41.422013 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:41.422020 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:41.422022 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:41.422582 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:41.422781 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:41.423591 (+ 810us) server_negotiation.cc:299] Negotiation successful
0812 09:45:41.423901 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":157,"threads_started":1}
W20260812 09:45:41.426683 23682 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f8aaf46e4a6d4531ab509eb45d22e2cd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 09:45:41.426785 23682 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 09:45:41.427085 23670 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 09:45:41.427963 23670 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 09:45:41.433535 23670 catalog_manager.cc:1347] Generated new cluster ID: 5feee61a67a142d8bd855e4f84785f79
I20260812 09:45:41.433597 23670 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 09:45:41.449762 23670 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 09:45:41.451090 23670 catalog_manager.cc:1504] Loading token signing keys...
I20260812 09:45:41.460737 23670 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f8aaf46e4a6d4531ab509eb45d22e2cd: Generated new TSK 0
I20260812 09:45:41.461423 23670 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.13.99.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.99.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.99.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/krb5kdc/oryx_127.13.99.1.keytab.
Entry for principal oryx/127.13.99.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/krb5kdc/oryx_127.13.99.1.keytab.
I20260812 09:45:41.495663 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.99.1:0
--local_ip_for_outbound_sockets=127.13.99.1
--webserver_interface=127.13.99.1
--webserver_port=0
--tserver_master_addrs=127.13.99.62:43093
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/krb5kdc/oryx_127.13.99.1.keytab
--principal=oryx/127.13.99.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:45813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:45:41.619385 23693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:45:41.619642 23693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:45:41.619685 23693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:45:41.624214 23693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:45:41.624284 23693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:45:41.624343 23693 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.99.1
I20260812 09:45:41.629747 23693 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:45813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/krb5kdc/oryx_127.13.99.1.keytab
--principal=oryx/127.13.99.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.99.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.99.1
--webserver_port=0
--tserver_master_addrs=127.13.99.62:43093
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.99.1
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:45:41.630792 23693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:45:41.631821 23693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 09:45:41.641814 23693 server_base.cc:1034] running on GCE node
W20260812 09:45:41.641849 23705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:41.641887 23700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:41.641881 23702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:45:41.643199 23693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:45:41.643762 23693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:45:41.644961 23693 hybrid_clock.cc:648] HybridClock initialized: now 1786527941644945 us; error 26 us; skew 500 ppm
Aug 12 09:45:41 dist-test-slave-jq4t krb5kdc[23544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527941, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.99.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:45:41.647797 23693 init.cc:377] Logged in from keytab as oryx/127.13.99.1@KRBTEST.COM (short username oryx)
I20260812 09:45:41.648795 23693 webserver.cc:460] Webserver started at http://127.13.99.1:46441/ using document root <none> and password file <none>
I20260812 09:45:41.649394 23693 fs_manager.cc:359] Metadata directory not provided
I20260812 09:45:41.649457 23693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:45:41.649674 23693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:45:41.651707 23693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/ts-0/data/instance:
uuid: "22cbcc6238b04ef196d61a4623d51639"
format_stamp: "Formatted at 2026-08-12 09:45:41 on dist-test-slave-jq4t"
server_key: "a494b14262578f0cfd435f5db85c1a63"
server_key_iv: "bfc70db1382ee4ab680bfc03b914029d"
server_key_version: "encryptionkey@0"
I20260812 09:45:41.652288 23693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/ts-0/wal/instance:
uuid: "22cbcc6238b04ef196d61a4623d51639"
format_stamp: "Formatted at 2026-08-12 09:45:41 on dist-test-slave-jq4t"
server_key: "a494b14262578f0cfd435f5db85c1a63"
server_key_iv: "bfc70db1382ee4ab680bfc03b914029d"
server_key_version: "encryptionkey@0"
I20260812 09:45:41.656018 23693 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260812 09:45:41.658458 23715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:41.659729 23693 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 09:45:41.659854 23693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/ts-0/wal
uuid: "22cbcc6238b04ef196d61a4623d51639"
format_stamp: "Formatted at 2026-08-12 09:45:41 on dist-test-slave-jq4t"
server_key: "a494b14262578f0cfd435f5db85c1a63"
server_key_iv: "bfc70db1382ee4ab680bfc03b914029d"
server_key_version: "encryptionkey@0"
I20260812 09:45:41.659978 23693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:45:41.671645 23693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:45:41.674538 23693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:45:41.674736 23693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:45:41.675312 23693 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 09:45:41.676163 23693 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:45:41.676211 23693 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:41.676281 23693 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:45:41.676313 23693 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:41.685336 23693 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.1:33935
I20260812 09:45:41.685379 23886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.1:33935 every 8 connection(s)
I20260812 09:45:41.686231 23693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/ts-0/data/info.pb
I20260812 09:45:41.693602 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 23693
I20260812 09:45:41.693751 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/ts-0/wal/instance
I20260812 09:45:41.694133 13708 external_mini_cluster.cc:1428] Setting key 8ebe9b68487da526d769757792763049
Aug 12 09:45:41 dist-test-slave-jq4t krb5kdc[23544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527941, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.99.1@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM
I20260812 09:45:41.700321 23658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:41.688065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.1:41981 (local address 127.13.99.62:43093)
0812 09:45:41.688238 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:41.688243 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:41.689115 (+ 872us) server_negotiation.cc:407] Connection header received
0812 09:45:41.690483 (+ 1368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:41.690486 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:41.690536 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:41.690616 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:41.691761 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:41.692300 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:41.693510 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:41.693781 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:41.693919 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:41.696670 (+ 2751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:41.696674 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:41.696677 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:41.696710 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:41.698167 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:41.698723 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:41.698727 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:41.698728 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:41.698806 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:41.699186 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:41.699189 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:41.699191 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:41.699376 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:41.699485 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:41.700023 (+ 538us) server_negotiation.cc:299] Negotiation successful
0812 09:45:41.700150 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260812 09:45:41.701099 23890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:41.688431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:43093 (local address 127.13.99.1:41981)
0812 09:45:41.688939 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:41.688999 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:41.690254 (+ 1255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:41.690782 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:41.690789 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:41.691144 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:45:41.691611 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:41.691625 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:41.692413 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:41.692416 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:41.693359 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:41.693373 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:41.693965 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:41.693970 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:41.694177 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:41.694860 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:45:41.694880 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:45:41.696450 (+ 1570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:45:41.698296 (+ 1846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:41.698302 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:41.698316 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:41.698590 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:41.698921 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:41.698926 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:41.698928 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:41.699078 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:41.699509 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:45:41.699515 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:45:41.699749 (+ 234us) client_negotiation.cc:769] Sending connection context
0812 09:45:41.699933 (+ 184us) client_negotiation.cc:240] Negotiation successful
0812 09:45:41.700252 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":110,"threads_started":1}
I20260812 09:45:41.702309 23887 heartbeater.cc:347] Connected to a master server at 127.13.99.62:43093
I20260812 09:45:41.702574 23887 heartbeater.cc:464] Registering TS with master...
I20260812 09:45:41.703186 23887 heartbeater.cc:511] Master 127.13.99.62:43093 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.13.99.2@KRBTEST.COM; defaulting to no policy
I20260812 09:45:41.704939 23606 ts_manager.cc:194] Registered new tserver with Master: 22cbcc6238b04ef196d61a4623d51639 (127.13.99.1:33935)
I20260812 09:45:41.706212 23606 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.99.1@KRBTEST.COM'} at 127.13.99.1:41981
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.99.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.99.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/krb5kdc/oryx_127.13.99.2.keytab.
Entry for principal oryx/127.13.99.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/krb5kdc/oryx_127.13.99.2.keytab.
I20260812 09:45:41.723842 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.99.2:0
--local_ip_for_outbound_sockets=127.13.99.2
--webserver_interface=127.13.99.2
--webserver_port=0
--tserver_master_addrs=127.13.99.62:43093
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/krb5kdc/oryx_127.13.99.2.keytab
--principal=oryx/127.13.99.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:45813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:45:41.837174 23893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:45:41.837422 23893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:45:41.837462 23893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:45:41.841483 23893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:45:41.841539 23893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:45:41.841595 23893 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.99.2
I20260812 09:45:41.846634 23893 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:45813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/krb5kdc/oryx_127.13.99.2.keytab
--principal=oryx/127.13.99.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.99.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.99.2
--webserver_port=0
--tserver_master_addrs=127.13.99.62:43093
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23893
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.99.2
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:45:41.847615 23893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:45:41.848695 23893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 09:45:41.858469 23893 server_base.cc:1034] running on GCE node
W20260812 09:45:41.858381 23904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:41.858387 23901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:41.858487 23902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:45:41.859015 23893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:45:41.859658 23893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:45:41.860847 23893 hybrid_clock.cc:648] HybridClock initialized: now 1786527941860815 us; error 44 us; skew 500 ppm
Aug 12 09:45:41 dist-test-slave-jq4t krb5kdc[23544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527941, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.99.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:45:41.863772 23893 init.cc:377] Logged in from keytab as oryx/127.13.99.2@KRBTEST.COM (short username oryx)
I20260812 09:45:41.864887 23893 webserver.cc:460] Webserver started at http://127.13.99.2:39009/ using document root <none> and password file <none>
I20260812 09:45:41.865476 23893 fs_manager.cc:359] Metadata directory not provided
I20260812 09:45:41.865572 23893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:45:41.865828 23893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:45:41.867872 23893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/ts-1/data/instance:
uuid: "433ef24880bd41ce9d2092d3748f7792"
format_stamp: "Formatted at 2026-08-12 09:45:41 on dist-test-slave-jq4t"
server_key: "622b55a38e01d16ef2d22fc994b06184"
server_key_iv: "3158c66a005ab6680bb66f082e69f6b0"
server_key_version: "encryptionkey@0"
I20260812 09:45:41.868459 23893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/ts-1/wal/instance:
uuid: "433ef24880bd41ce9d2092d3748f7792"
format_stamp: "Formatted at 2026-08-12 09:45:41 on dist-test-slave-jq4t"
server_key: "622b55a38e01d16ef2d22fc994b06184"
server_key_iv: "3158c66a005ab6680bb66f082e69f6b0"
server_key_version: "encryptionkey@0"
I20260812 09:45:41.872339 23893 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260812 09:45:41.874825 23915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:41.876031 23893 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 09:45:41.876165 23893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/ts-1/wal
uuid: "433ef24880bd41ce9d2092d3748f7792"
format_stamp: "Formatted at 2026-08-12 09:45:41 on dist-test-slave-jq4t"
server_key: "622b55a38e01d16ef2d22fc994b06184"
server_key_iv: "3158c66a005ab6680bb66f082e69f6b0"
server_key_version: "encryptionkey@0"
I20260812 09:45:41.876272 23893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:45:41.891088 23893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:45:41.893966 23893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:45:41.894160 23893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:45:41.894714 23893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 09:45:41.895515 23893 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:45:41.895591 23893 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:41.895658 23893 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:45:41.895705 23893 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:41.904234 23893 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.2:46293
I20260812 09:45:41.904310 24079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.2:46293 every 8 connection(s)
I20260812 09:45:41.905206 23893 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/ts-1/data/info.pb
I20260812 09:45:41.910847 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 23893
I20260812 09:45:41.911027 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/ts-1/wal/instance
I20260812 09:45:41.911334 13708 external_mini_cluster.cc:1428] Setting key 48017f89a42bfb44d8f805e3be9a4bae
Aug 12 09:45:41 dist-test-slave-jq4t krb5kdc[23544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527941, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.99.2@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM
I20260812 09:45:41.919759 23658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:41.907007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.2:41239 (local address 127.13.99.62:43093)
0812 09:45:41.907179 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:41.907183 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:41.908002 (+ 819us) server_negotiation.cc:407] Connection header received
0812 09:45:41.909415 (+ 1413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:41.909418 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:41.909468 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:41.909546 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:41.910693 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:41.911349 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:41.912802 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:41.913117 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:41.913232 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:41.915892 (+ 2660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:41.915896 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:41.915898 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:41.915926 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:41.917433 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:41.917994 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:41.917999 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:41.918002 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:41.918109 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:41.918463 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:41.918468 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:41.918471 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:41.918750 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:41.918883 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:41.919366 (+ 483us) server_negotiation.cc:299] Negotiation successful
0812 09:45:41.919513 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260812 09:45:41.920370 24084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:41.907378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:43093 (local address 127.13.99.2:41239)
0812 09:45:41.907827 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:41.907869 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:41.909175 (+ 1306us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:41.909714 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:41.909721 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:41.910064 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:45:41.910522 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:41.910545 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:41.911478 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:41.911482 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:41.912653 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:41.912663 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:41.913283 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:41.913287 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:41.913439 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:41.914055 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:45:41.914071 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:45:41.915701 (+ 1630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:45:41.917568 (+ 1867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:41.917575 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:41.917590 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:41.917851 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:41.918221 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:41.918223 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:41.918224 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:41.918356 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:41.918864 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:45:41.918869 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:45:41.919110 (+ 241us) client_negotiation.cc:769] Sending connection context
0812 09:45:41.919267 (+ 157us) client_negotiation.cc:240] Negotiation successful
0812 09:45:41.919517 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":114,"threads_started":1}
I20260812 09:45:41.921684 24081 heartbeater.cc:347] Connected to a master server at 127.13.99.62:43093
I20260812 09:45:41.921986 24081 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for oryx/127.13.99.3@KRBTEST.COM; defaulting to no policy
I20260812 09:45:41.922612 24081 heartbeater.cc:511] Master 127.13.99.62:43093 requested a full tablet report, sending...
I20260812 09:45:41.923734 23606 ts_manager.cc:194] Registered new tserver with Master: 433ef24880bd41ce9d2092d3748f7792 (127.13.99.2:46293)
I20260812 09:45:41.924314 23606 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.99.2@KRBTEST.COM'} at 127.13.99.2:41239
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.99.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.99.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/krb5kdc/oryx_127.13.99.3.keytab.
Entry for principal oryx/127.13.99.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/krb5kdc/oryx_127.13.99.3.keytab.
I20260812 09:45:41.939723 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.99.3:0
--local_ip_for_outbound_sockets=127.13.99.3
--webserver_interface=127.13.99.3
--webserver_port=0
--tserver_master_addrs=127.13.99.62:43093
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/krb5kdc/oryx_127.13.99.3.keytab
--principal=oryx/127.13.99.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:45813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:45:42.053620 24087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:45:42.053858 24087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:45:42.053900 24087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:45:42.058182 24087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:45:42.058243 24087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:45:42.058302 24087 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.99.3
I20260812 09:45:42.063750 24087 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:45813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/krb5kdc/oryx_127.13.99.3.keytab
--principal=oryx/127.13.99.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.99.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.99.3
--webserver_port=0
--tserver_master_addrs=127.13.99.62:43093
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.99.3
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:45:42.064978 24087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:45:42.066145 24087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:45:42.076387 24097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:42.076395 24094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:42.076406 24099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:45:42.076668 24087 server_base.cc:1034] running on GCE node
I20260812 09:45:42.077142 24087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:45:42.077786 24087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:45:42.078989 24087 hybrid_clock.cc:648] HybridClock initialized: now 1786527942078965 us; error 36 us; skew 500 ppm
Aug 12 09:45:42 dist-test-slave-jq4t krb5kdc[23544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527942, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.99.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:45:42.081969 24087 init.cc:377] Logged in from keytab as oryx/127.13.99.3@KRBTEST.COM (short username oryx)
I20260812 09:45:42.082897 24087 webserver.cc:460] Webserver started at http://127.13.99.3:36725/ using document root <none> and password file <none>
I20260812 09:45:42.083477 24087 fs_manager.cc:359] Metadata directory not provided
I20260812 09:45:42.083583 24087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:45:42.083808 24087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:45:42.086035 24087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/ts-2/data/instance:
uuid: "022e82db4dab49d29d2f320818a7f2f8"
format_stamp: "Formatted at 2026-08-12 09:45:42 on dist-test-slave-jq4t"
server_key: "e033f637072b022d2fe4385147fb0811"
server_key_iv: "82c7e200e0031c7bf6b2782c654a6bc7"
server_key_version: "encryptionkey@0"
I20260812 09:45:42.086628 24087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/ts-2/wal/instance:
uuid: "022e82db4dab49d29d2f320818a7f2f8"
format_stamp: "Formatted at 2026-08-12 09:45:42 on dist-test-slave-jq4t"
server_key: "e033f637072b022d2fe4385147fb0811"
server_key_iv: "82c7e200e0031c7bf6b2782c654a6bc7"
server_key_version: "encryptionkey@0"
I20260812 09:45:42.090508 24087 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 09:45:42.093008 24107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:42.094233 24087 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 09:45:42.094372 24087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/ts-2/wal
uuid: "022e82db4dab49d29d2f320818a7f2f8"
format_stamp: "Formatted at 2026-08-12 09:45:42 on dist-test-slave-jq4t"
server_key: "e033f637072b022d2fe4385147fb0811"
server_key_iv: "82c7e200e0031c7bf6b2782c654a6bc7"
server_key_version: "encryptionkey@0"
I20260812 09:45:42.094482 24087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:45:42.113783 24087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:45:42.116765 24087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:45:42.116971 24087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:45:42.117574 24087 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 09:45:42.118415 24087 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:45:42.118492 24087 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:42.118572 24087 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:45:42.118625 24087 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:42.128146 24087 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.3:44373
I20260812 09:45:42.128180 24263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.3:44373 every 8 connection(s)
I20260812 09:45:42.129050 24087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/ts-2/data/info.pb
I20260812 09:45:42.136703 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 24087
I20260812 09:45:42.136834 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/ts-2/wal/instance
I20260812 09:45:42.137197 13708 external_mini_cluster.cc:1428] Setting key ca19dc1d2d01280705ce127b6dd1223b
Aug 12 09:45:42 dist-test-slave-jq4t krb5kdc[23544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527942, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.99.3@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM
I20260812 09:45:42.142820 23658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:42.130789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.3:51167 (local address 127.13.99.62:43093)
0812 09:45:42.130970 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:42.130975 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:42.131798 (+ 823us) server_negotiation.cc:407] Connection header received
0812 09:45:42.133200 (+ 1402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:42.133203 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:42.133252 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:42.133330 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:42.134449 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:42.135045 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:42.136145 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:42.136359 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:42.136436 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:42.139173 (+ 2737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:42.139179 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:42.139186 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:42.139210 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:42.140771 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:42.141299 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:42.141307 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:42.141308 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:42.141382 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:42.141742 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:42.141746 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:42.141748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:42.141937 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:42.142049 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:42.142539 (+ 490us) server_negotiation.cc:299] Negotiation successful
0812 09:45:42.142653 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260812 09:45:42.143565 24269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:42.131195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:43093 (local address 127.13.99.3:51167)
0812 09:45:42.131644 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:42.131686 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:42.132973 (+ 1287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:42.133495 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:42.133501 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:42.133847 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:45:42.134309 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:42.134322 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:42.135159 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:42.135162 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:42.136037 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:42.136047 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:42.136554 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:42.136562 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:42.136786 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:42.137535 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:45:42.137552 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:45:42.139005 (+ 1453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:45:42.140902 (+ 1897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:42.140909 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:42.140922 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:42.141180 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:42.141495 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:42.141498 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:42.141500 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:42.141639 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:42.142050 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:45:42.142058 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:45:42.142291 (+ 233us) client_negotiation.cc:769] Sending connection context
0812 09:45:42.142483 (+ 192us) client_negotiation.cc:240] Negotiation successful
0812 09:45:42.142787 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":106,"threads_started":1}
I20260812 09:45:42.144675 24264 heartbeater.cc:347] Connected to a master server at 127.13.99.62:43093
I20260812 09:45:42.144922 24264 heartbeater.cc:464] Registering TS with master...
I20260812 09:45:42.145401 24264 heartbeater.cc:511] Master 127.13.99.62:43093 requested a full tablet report, sending...
I20260812 09:45:42.146421 23606 ts_manager.cc:194] Registered new tserver with Master: 022e82db4dab49d29d2f320818a7f2f8 (127.13.99.3:44373)
I20260812 09:45:42.146996 23606 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.99.3@KRBTEST.COM'} at 127.13.99.3:51167
I20260812 09:45:42.150951 13708 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.13.99.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.99.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.99.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/krb5kdc/oryx_127.13.99.61.keytab.
Entry for principal oryx/127.13.99.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/krb5kdc/oryx_127.13.99.61.keytab.
I20260812 09:45:42.180034 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.61:36161
--webserver_interface=127.13.99.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/krb5kdc/oryx_127.13.99.61.keytab
--principal=oryx/127.13.99.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:45813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.99.62:43093,127.13.99.61:36161
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:45:42.293406 24274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:45:42.293684 24274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:45:42.293756 24274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:45:42.297804 24274 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 09:45:42.297878 24274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:45:42.297914 24274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:45:42.297973 24274 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 09:45:42.298012 24274 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 09:45:42.303198 24274 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:45813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.13.99.62:43093,127.13.99.61:36161
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/krb5kdc/oryx_127.13.99.61.keytab
--principal=oryx/127.13.99.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.61:36161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.13.99.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24274
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
W20260812 09:45:42.303443 24274 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.13.99.62:43093,127.13.99.61:36161'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260812 09:45:42.304281 24274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:45:42.305351 24274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:45:42.313611 24282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:42.313656 24281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:45:42.313709 24274 server_base.cc:1034] running on GCE node
W20260812 09:45:42.313647 24284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:45:42.314393 24274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:45:42.315452 24274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:45:42.316658 24274 hybrid_clock.cc:648] HybridClock initialized: now 1786527942316637 us; error 30 us; skew 500 ppm
Aug 12 09:45:42 dist-test-slave-jq4t krb5kdc[23544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527942, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.99.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:45:42.319563 24274 init.cc:377] Logged in from keytab as oryx/127.13.99.61@KRBTEST.COM (short username oryx)
I20260812 09:45:42.320416 24274 webserver.cc:460] Webserver started at http://127.13.99.61:35893/ using document root <none> and password file <none>
I20260812 09:45:42.321081 24274 fs_manager.cc:359] Metadata directory not provided
I20260812 09:45:42.321148 24274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:45:42.321317 24274 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:45:42.323585 24274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/master-1/data/instance:
uuid: "cf7b481d137c42268998cd7a08f2d65e"
format_stamp: "Formatted at 2026-08-12 09:45:42 on dist-test-slave-jq4t"
server_key: "e94f6d1cd76cee057b4831548a7dc59c"
server_key_iv: "5e257dd0f24af2b731a205f7a771d625"
server_key_version: "encryptionkey@0"
I20260812 09:45:42.324115 24274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/master-1/wal/instance:
uuid: "cf7b481d137c42268998cd7a08f2d65e"
format_stamp: "Formatted at 2026-08-12 09:45:42 on dist-test-slave-jq4t"
server_key: "e94f6d1cd76cee057b4831548a7dc59c"
server_key_iv: "5e257dd0f24af2b731a205f7a771d625"
server_key_version: "encryptionkey@0"
I20260812 09:45:42.327785 24274 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260812 09:45:42.330175 24300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:42.331405 24274 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260812 09:45:42.331540 24274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/master-1/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/master-1/wal
uuid: "cf7b481d137c42268998cd7a08f2d65e"
format_stamp: "Formatted at 2026-08-12 09:45:42 on dist-test-slave-jq4t"
server_key: "e94f6d1cd76cee057b4831548a7dc59c"
server_key_iv: "5e257dd0f24af2b731a205f7a771d625"
server_key_version: "encryptionkey@0"
I20260812 09:45:42.331640 24274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:45:42.342972 24274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:45:42.346211 24274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:45:42.346421 24274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:45:42.353405 24274 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.61:36161
I20260812 09:45:42.353684 24375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.61:36161 every 8 connection(s)
I20260812 09:45:42.354426 24274 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/master-1/data/info.pb
I20260812 09:45:42.356741 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 24274
I20260812 09:45:42.357404 24376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:45:42.362187 24376 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.99.62" port: 43093 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 09:45:42 dist-test-slave-jq4t krb5kdc[23544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527941, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.13.99.61@KRBTEST.COM
Aug 12 09:45:42 dist-test-slave-jq4t krb5kdc[23544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527942, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.99.61@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM
I20260812 09:45:42.372594 24380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:42.358414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55950 (local address 127.13.99.61:36161)
0812 09:45:42.358837 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:42.358864 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:42.358901 (+ 37us) server_negotiation.cc:407] Connection header received
0812 09:45:42.359560 (+ 659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:42.359577 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:42.359910 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:42.360193 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:42.361173 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:42.362193 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:42.363379 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:42.364022 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:42.364245 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:42.366969 (+ 2724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:42.366975 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:42.366984 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:42.367010 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:42.369054 (+ 2044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:42.369587 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:42.369592 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:42.369596 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:42.369712 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:42.370087 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:42.370090 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:42.370091 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:42.370536 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:42.370790 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:42.371219 (+ 429us) server_negotiation.cc:299] Negotiation successful
0812 09:45:42.371519 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":151,"threads_started":1}
I20260812 09:45:42.374154 23658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:42.364287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54478 (local address 127.13.99.62:43093)
0812 09:45:42.364426 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:42.364431 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:42.365096 (+ 665us) server_negotiation.cc:407] Connection header received
0812 09:45:42.365180 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:42.365183 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:42.365249 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:42.365348 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:42.366009 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:42.366584 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:42.367665 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:42.367888 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:42.368005 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:42.370489 (+ 2484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:42.370493 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:42.370496 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:42.370522 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:42.372206 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:42.372768 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:42.372771 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:42.372773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:42.372870 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:42.373176 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:42.373180 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:42.373181 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:42.373457 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:42.373576 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:42.373943 (+ 367us) server_negotiation.cc:299] Negotiation successful
0812 09:45:42.374045 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260812 09:45:42.374199 24384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:42.364209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:43093 (local address 127.0.0.1:54478)
0812 09:45:42.364962 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:42.365004 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:42.365107 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:42.365396 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:42.365400 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:42.365732 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:45:42.365855 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:42.365861 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:42.366722 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:42.366726 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:42.367529 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:42.367540 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:42.368002 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:42.368006 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:42.368126 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:42.368643 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:45:42.368655 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:45:42.370339 (+ 1684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:45:42.372320 (+ 1981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:42.372328 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:42.372333 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:42.372626 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:42.372978 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:42.372981 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:42.372982 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:42.373067 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:42.373569 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:45:42.373574 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:45:42.373717 (+ 143us) client_negotiation.cc:769] Sending connection context
0812 09:45:42.373855 (+ 138us) client_negotiation.cc:240] Negotiation successful
0812 09:45:42.374021 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":95,"threads_started":1}
I20260812 09:45:42.375406 24376 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.99.61" port: 36161 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 09:45:42 dist-test-slave-jq4t krb5kdc[23544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527942, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.99.61@KRBTEST.COM for oryx/127.13.99.61@KRBTEST.COM
I20260812 09:45:42.383399 24384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:42.375897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.61:36161 (local address 127.0.0.1:55952)
0812 09:45:42.376008 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:42.376022 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:42.376100 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:42.376393 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:42.376395 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:42.376585 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:45:42.376673 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:42.376678 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:42.377796 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:42.377799 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:42.378312 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:42.378320 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:42.378728 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:42.378730 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:42.378799 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:42.379301 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:45:42.379313 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:45:42.380619 (+ 1306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:45:42.382143 (+ 1524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:42.382146 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:42.382147 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:42.382349 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:42.382609 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:42.382611 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:42.382612 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:42.382689 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:42.383003 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:45:42.383006 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:45:42.383079 (+ 73us) client_negotiation.cc:769] Sending connection context
0812 09:45:42.383159 (+ 80us) client_negotiation.cc:240] Negotiation successful
0812 09:45:42.383262 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260812 09:45:42.383433 24380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:42.376065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55952 (local address 127.13.99.61:36161)
0812 09:45:42.376209 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:42.376212 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:42.376225 (+ 13us) server_negotiation.cc:407] Connection header received
0812 09:45:42.376255 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:42.376257 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:42.376289 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:42.376365 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:42.376776 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:42.377676 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:42.378418 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:42.378633 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:42.378712 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:42.380816 (+ 2104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:42.380820 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:42.380824 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:42.380852 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:42.382006 (+ 1154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:42.382443 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:42.382445 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:42.382447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:42.382525 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:42.382768 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:42.382770 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:42.382771 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:42.382923 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:42.382995 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:42.383188 (+ 193us) server_negotiation.cc:299] Negotiation successful
0812 09:45:42.383315 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":73}
I20260812 09:45:42.386461 24376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf7b481d137c42268998cd7a08f2d65e: Bootstrap starting.
I20260812 09:45:42.388819 24376 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf7b481d137c42268998cd7a08f2d65e: Neither blocks nor log segments found. Creating new log.
I20260812 09:45:42.389559 24376 log.cc:824] T 00000000000000000000000000000000 P cf7b481d137c42268998cd7a08f2d65e: Log is configured to *not* fsync() on all Append() calls
I20260812 09:45:42.391618 24376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf7b481d137c42268998cd7a08f2d65e: No bootstrap required, opened a new log
I20260812 09:45:42.394793 24376 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf7b481d137c42268998cd7a08f2d65e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8aaf46e4a6d4531ab509eb45d22e2cd" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 43093 } } peers { permanent_uuid: "cf7b481d137c42268998cd7a08f2d65e" member_type: VOTER last_known_addr { host: "127.13.99.61" port: 36161 } }
I20260812 09:45:42.394990 24376 raft_consensus.cc:374] T 00000000000000000000000000000000 P cf7b481d137c42268998cd7a08f2d65e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:45:42.395072 24376 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf7b481d137c42268998cd7a08f2d65e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf7b481d137c42268998cd7a08f2d65e, State: Initialized, Role: FOLLOWER
I20260812 09:45:42.395531 24376 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf7b481d137c42268998cd7a08f2d65e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8aaf46e4a6d4531ab509eb45d22e2cd" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 43093 } } peers { permanent_uuid: "cf7b481d137c42268998cd7a08f2d65e" member_type: VOTER last_known_addr { host: "127.13.99.61" port: 36161 } }
I20260812 09:45:42.396044 24386 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf7b481d137c42268998cd7a08f2d65e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8aaf46e4a6d4531ab509eb45d22e2cd" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 43093 } } peers { permanent_uuid: "cf7b481d137c42268998cd7a08f2d65e" member_type: VOTER last_known_addr { host: "127.13.99.61" port: 36161 } } }
I20260812 09:45:42.396176 24386 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf7b481d137c42268998cd7a08f2d65e [sys.catalog]: This master's current role is: FOLLOWER
I20260812 09:45:42.396790 24376 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf7b481d137c42268998cd7a08f2d65e [sys.catalog]: configured and running, proceeding with master startup.
W20260812 09:45:42.402400 24405 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cf7b481d137c42268998cd7a08f2d65e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 09:45:42.402536 24405 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 09:45:42.405212 24274 master_runner.cc:428] Detected that this master cf7b481d137c42268998cd7a08f2d65e is joining an existing cluster
I20260812 09:45:42.405279 24274 master_runner.cc:432] Initiating AddMaster RPC to add 127.13.99.61:36161
I20260812 09:45:42.417330 24380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:42.408436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55966 (local address 127.13.99.61:36161)
0812 09:45:42.408565 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:42.408569 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:42.408833 (+ 264us) server_negotiation.cc:407] Connection header received
0812 09:45:42.408956 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:42.408959 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:42.408989 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:42.409027 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:42.409559 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:42.410272 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:42.411384 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:42.411720 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:42.411806 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:42.413680 (+ 1874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:42.413684 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:42.413686 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:42.413704 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:42.415215 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:42.415636 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:42.415643 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:42.415647 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:42.415750 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:42.416284 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:42.416287 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:42.416288 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:42.416601 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:42.416709 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:42.417011 (+ 302us) server_negotiation.cc:299] Negotiation successful
0812 09:45:42.417156 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":62}
I20260812 09:45:42.417423 23658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:42.407695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54490 (local address 127.13.99.62:43093)
0812 09:45:42.407847 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:42.407852 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:42.408103 (+ 251us) server_negotiation.cc:407] Connection header received
0812 09:45:42.408215 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:42.408218 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:42.408271 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:42.408347 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:42.409079 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:42.409926 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:42.411219 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:42.411557 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:42.411636 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:42.413425 (+ 1789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:42.413429 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:42.413432 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:42.413459 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:42.415642 (+ 2183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:42.416071 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:42.416076 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:42.416078 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:42.416163 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:42.416525 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:42.416529 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:42.416531 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:42.416754 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:42.416868 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:42.417154 (+ 286us) server_negotiation.cc:299] Negotiation successful
0812 09:45:42.417269 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260812 09:45:42.417738 24416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:42.408433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.61:36161 (local address 127.0.0.1:55966)
0812 09:45:42.408765 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:42.408777 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:42.408878 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:42.409126 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:42.409128 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:42.409334 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:45:42.409462 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:42.409468 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:42.410390 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:42.410393 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:42.411271 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:42.411280 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:42.411937 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:42.411940 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:42.412039 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:42.413092 (+ 1053us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:45:42.413105 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:45:42.413555 (+ 450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:45:42.415332 (+ 1777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:42.415337 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:42.415339 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:42.415537 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:42.415838 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:42.415841 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:42.415843 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:42.415955 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:42.416697 (+ 742us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:45:42.416700 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:45:42.416788 (+ 88us) client_negotiation.cc:769] Sending connection context
0812 09:45:42.416949 (+ 161us) client_negotiation.cc:240] Negotiation successful
0812 09:45:42.417601 (+ 652us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":110,"threads_started":1}
I20260812 09:45:42.417778 24415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:42.407624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:43093 (local address 127.0.0.1:54490)
0812 09:45:42.408005 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:42.408020 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:42.408117 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:42.408516 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:42.408519 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:42.408758 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:45:42.408922 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:42.408933 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:42.410075 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:42.410078 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:42.411072 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:42.411083 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:42.411842 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:42.411845 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:42.411950 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:42.412751 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:45:42.412767 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:45:42.413261 (+ 494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:45:42.415749 (+ 2488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:42.415752 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:42.415754 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:42.415969 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:42.416272 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:42.416275 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:42.416277 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:42.416382 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:42.416857 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:45:42.416861 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:45:42.416955 (+ 94us) client_negotiation.cc:769] Sending connection context
0812 09:45:42.417096 (+ 141us) client_negotiation.cc:240] Negotiation successful
0812 09:45:42.417212 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":108,"threads_started":1}
Aug 12 09:45:42 dist-test-slave-jq4t krb5kdc[23544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527941, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.99.62@KRBTEST.COM for oryx/127.13.99.61@KRBTEST.COM
I20260812 09:45:42.434878 24380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:42.423157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55974 (local address 127.13.99.61:36161)
0812 09:45:42.423273 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:42.423277 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:42.424888 (+ 1611us) server_negotiation.cc:407] Connection header received
0812 09:45:42.425133 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:42.425137 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:42.425193 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:42.425288 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:42.426144 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:42.426631 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:42.427639 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:42.427851 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:42.427962 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:42.430817 (+ 2855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:42.430821 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:42.430824 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:42.430850 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:42.432672 (+ 1822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:42.433304 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:42.433308 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:42.433309 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:42.433398 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:42.433745 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:42.433749 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:42.433751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:42.434012 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:42.434116 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:42.434573 (+ 457us) server_negotiation.cc:299] Negotiation successful
0812 09:45:42.434690 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260812 09:45:42.434962 24417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:42.423066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.61:36161 (local address 127.0.0.1:55974)
0812 09:45:42.424754 (+ 1688us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:42.424806 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:42.424953 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:42.425345 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:42.425352 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:42.425831 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:45:42.425975 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:42.425983 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:42.426751 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:42.426754 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:42.427507 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:42.427518 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:42.427966 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:42.427970 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:42.428092 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:42.428714 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:45:42.428734 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:45:42.430650 (+ 1916us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:45:42.432795 (+ 2145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:42.432810 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:42.432819 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:42.433184 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:42.433509 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:42.433515 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:42.433519 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:42.433635 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:42.434145 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:45:42.434155 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:45:42.434332 (+ 177us) client_negotiation.cc:769] Sending connection context
0812 09:45:42.434507 (+ 175us) client_negotiation.cc:240] Negotiation successful
0812 09:45:42.434737 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1514,"thread_start_us":94,"threads_started":1}
WARNING: no policy specified for oryx/127.13.99.60@KRBTEST.COM; defaulting to no policy
I20260812 09:45:42.437129 23606 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.13.99.61:36161: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "cf7b481d137c42268998cd7a08f2d65e"
member_type: NON_VOTER
last_known_addr {
host: "127.13.99.61"
port: 36161
}
attrs {
promote: true
}
}
dest_uuid: "f8aaf46e4a6d4531ab509eb45d22e2cd"
cas_config_opid_index: -1
I20260812 09:45:42.437932 23606 consensus_queue.cc:237] T 00000000000000000000000000000000 P f8aaf46e4a6d4531ab509eb45d22e2cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f8aaf46e4a6d4531ab509eb45d22e2cd" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 43093 } } peers { permanent_uuid: "cf7b481d137c42268998cd7a08f2d65e" member_type: NON_VOTER last_known_addr { host: "127.13.99.61" port: 36161 } attrs { promote: true } }
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.99.60@KRBTEST.COM" created.
I20260812 09:45:42.439817 24421 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f8aaf46e4a6d4531ab509eb45d22e2cd [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER cf7b481d137c42268998cd7a08f2d65e (127.13.99.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f8aaf46e4a6d4531ab509eb45d22e2cd" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 43093 } } peers { permanent_uuid: "cf7b481d137c42268998cd7a08f2d65e" member_type: NON_VOTER last_known_addr { host: "127.13.99.61" port: 36161 } attrs { promote: true } } }
I20260812 09:45:42.440610 24338 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf7b481d137c42268998cd7a08f2d65e [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:45:42.441653 24421 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.13.99.61:36161
I20260812 09:45:42.441906 24420 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8aaf46e4a6d4531ab509eb45d22e2cd [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f8aaf46e4a6d4531ab509eb45d22e2cd" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f8aaf46e4a6d4531ab509eb45d22e2cd" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 43093 } } peers { permanent_uuid: "cf7b481d137c42268998cd7a08f2d65e" member_type: NON_VOTER last_known_addr { host: "127.13.99.61" port: 36161 } attrs { promote: true } } }
I20260812 09:45:42.442056 24338 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cf7b481d137c42268998cd7a08f2d65e [term 1 FOLLOWER]: Refusing update from remote peer f8aaf46e4a6d4531ab509eb45d22e2cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260812 09:45:42.442090 24420 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8aaf46e4a6d4531ab509eb45d22e2cd [sys.catalog]: This master's current role is: LEADER
I20260812 09:45:42.442760 24274 master.cc:537] Master@127.13.99.61:36161 shutting down...
I20260812 09:45:42.442790 24420 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f8aaf46e4a6d4531ab509eb45d22e2cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf7b481d137c42268998cd7a08f2d65e" member_type: NON_VOTER last_known_addr { host: "127.13.99.61" port: 36161 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 09:45:42.446717 23588 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260812 09:45:42.447569 24274 raft_consensus.cc:2229] T 00000000000000000000000000000000 P cf7b481d137c42268998cd7a08f2d65e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 09:45:42.447870 24274 raft_consensus.cc:2258] T 00000000000000000000000000000000 P cf7b481d137c42268998cd7a08f2d65e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 09:45:42.447978 24274 tablet_replica.cc:330] T 00000000000000000000000000000000 P cf7b481d137c42268998cd7a08f2d65e: stopping tablet replica
W20260812 09:45:42.448875 23588 consensus_peers.cc:487] T 00000000000000000000000000000000 P f8aaf46e4a6d4531ab509eb45d22e2cd -> Peer cf7b481d137c42268998cd7a08f2d65e (127.13.99.61:36161): Couldn't send request to peer cf7b481d137c42268998cd7a08f2d65e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260812 09:45:42.463171 24274 master.cc:552] Master@127.13.99.61:36161 shutdown complete.
I20260812 09:45:42.463241 24274 master_runner.cc:305] Clearing existing system tablet
I20260812 09:45:42.463478 24274 fs_manager.cc:359] Metadata directory not provided
I20260812 09:45:42.463555 24274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:45:42.465312 24274 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 09:45:42.466559 24428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:42.466750 24274 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:45:42.466907 24274 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 09:45:42.466948 24274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/master-1/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/master-1/wal
uuid: "cf7b481d137c42268998cd7a08f2d65e"
format_stamp: "Formatted at 2026-08-12 09:45:42 on dist-test-slave-jq4t"
server_key: "e94f6d1cd76cee057b4831548a7dc59c"
server_key_iv: "5e257dd0f24af2b731a205f7a771d625"
server_key_version: "encryptionkey@0"
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.99.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/krb5kdc/oryx_127.13.99.60.keytab.
Entry for principal oryx/127.13.99.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/krb5kdc/oryx_127.13.99.60.keytab.
I20260812 09:45:42.469398 24274 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P cf7b481d137c42268998cd7a08f2d65e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 09:45:42.469480 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.60:38095
--webserver_interface=127.13.99.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/krb5kdc/oryx_127.13.99.60.keytab
--principal=oryx/127.13.99.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:45813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.99.62:43093,127.13.99.61:36161,127.13.99.60:38095
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 09:45:42.471720 24274 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P cf7b481d137c42268998cd7a08f2d65e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260812 09:45:42.471827 24274 log.cc:1196] T 00000000000000000000000000000000 P cf7b481d137c42268998cd7a08f2d65e: Deleting WAL directory at /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260812 09:45:42.472103 24274 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P cf7b481d137c42268998cd7a08f2d65e: Deleting consensus metadata
I20260812 09:45:42.472294 24274 master_runner.cc:315] Copying system tablet from 127.13.99.62:43093
I20260812 09:45:42.473766 24274 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P cf7b481d137c42268998cd7a08f2d65e: tablet copy: Beginning tablet copy session from remote peer at address 127.13.99.62:43093
I20260812 09:45:42.481967 23658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:42.474468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54494 (local address 127.13.99.62:43093)
0812 09:45:42.474618 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:42.474623 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:42.474841 (+ 218us) server_negotiation.cc:407] Connection header received
0812 09:45:42.475002 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:42.475006 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:42.475069 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:42.475161 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:42.475628 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:42.476171 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:42.477092 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:42.477326 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:42.477442 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:42.478610 (+ 1168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:42.478614 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:42.478617 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:42.478643 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:42.480245 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:42.480755 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:42.480758 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:42.480760 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:42.480843 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:42.481136 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:42.481139 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:42.481141 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:42.481326 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:42.481435 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:42.481709 (+ 274us) server_negotiation.cc:299] Negotiation successful
0812 09:45:42.481822 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260812 09:45:42.481978 24440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:42.474399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:43093 (local address 127.0.0.1:54494)
0812 09:45:42.474743 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:42.474755 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:42.474896 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:42.475193 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:42.475196 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:42.475392 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:45:42.475498 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:42.475504 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:42.476289 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:42.476294 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:42.476961 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:42.476971 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:42.477449 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:42.477453 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:42.477562 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:42.478119 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:45:42.478131 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:45:42.478459 (+ 328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:45:42.480349 (+ 1890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:42.480355 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:42.480359 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:42.480653 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:42.480939 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:42.480943 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:42.480945 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:42.481036 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:42.481425 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:45:42.481428 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:45:42.481540 (+ 112us) client_negotiation.cc:769] Sending connection context
0812 09:45:42.481651 (+ 111us) client_negotiation.cc:240] Negotiation successful
0812 09:45:42.481795 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":86,"threads_started":1}
I20260812 09:45:42.482861 23633 tablet_copy_service.cc:140] P f8aaf46e4a6d4531ab509eb45d22e2cd: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer cf7b481d137c42268998cd7a08f2d65e ({username='oryx', principal='oryx/127.13.99.61@KRBTEST.COM'} at 127.0.0.1:54494)
I20260812 09:45:42.483057 23633 tablet_copy_service.cc:161] P f8aaf46e4a6d4531ab509eb45d22e2cd: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer cf7b481d137c42268998cd7a08f2d65e at {username='oryx', principal='oryx/127.13.99.61@KRBTEST.COM'} at 127.0.0.1:54494: session id = cf7b481d137c42268998cd7a08f2d65e-00000000000000000000000000000000
I20260812 09:45:42.484791 23633 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P f8aaf46e4a6d4531ab509eb45d22e2cd: Tablet Copy: opened 0 blocks and 1 log segments
I20260812 09:45:42.486367 24274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260812 09:45:42.488205 24274 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P cf7b481d137c42268998cd7a08f2d65e: tablet copy: Starting download of 0 data blocks...
I20260812 09:45:42.488408 24274 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P cf7b481d137c42268998cd7a08f2d65e: tablet copy: Starting download of 1 WAL segments...
I20260812 09:45:42.490082 24274 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P cf7b481d137c42268998cd7a08f2d65e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260812 09:45:42.491708 23633 tablet_copy_service.cc:342] P f8aaf46e4a6d4531ab509eb45d22e2cd: Request end of tablet copy session cf7b481d137c42268998cd7a08f2d65e-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.13.99.61@KRBTEST.COM'} at 127.0.0.1:54494
I20260812 09:45:42.491824 23633 tablet_copy_service.cc:433] P f8aaf46e4a6d4531ab509eb45d22e2cd: ending tablet copy session cf7b481d137c42268998cd7a08f2d65e-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer cf7b481d137c42268998cd7a08f2d65e
I20260812 09:45:42.499001 24274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:45:42.500694 24441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:42.500712 24446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:45:42.501111 24274 server_base.cc:1034] running on GCE node
W20260812 09:45:42.504623 24443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:45:42.504868 24274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:45:42.505259 24274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:45:42.506422 24274 hybrid_clock.cc:648] HybridClock initialized: now 1786527942506411 us; error 24 us; skew 500 ppm
Aug 12 09:45:42 dist-test-slave-jq4t krb5kdc[23544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527942, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.99.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:45:42.507899 24274 init.cc:377] Logged in from keytab as oryx/127.13.99.61@KRBTEST.COM (short username oryx)
I20260812 09:45:42.508392 24274 webserver.cc:460] Webserver started at http://127.13.99.61:34361/ using document root <none> and password file <none>
I20260812 09:45:42.508605 24274 fs_manager.cc:359] Metadata directory not provided
I20260812 09:45:42.508665 24274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:45:42.510073 24274 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 09:45:42.511011 24455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:42.511245 24274 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 09:45:42.511338 24274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/master-1/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/master-1/wal
uuid: "cf7b481d137c42268998cd7a08f2d65e"
format_stamp: "Formatted at 2026-08-12 09:45:42 on dist-test-slave-jq4t"
server_key: "e94f6d1cd76cee057b4831548a7dc59c"
server_key_iv: "5e257dd0f24af2b731a205f7a771d625"
server_key_version: "encryptionkey@0"
I20260812 09:45:42.511422 24274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:45:42.522990 24274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:45:42.525023 24274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:45:42.530280 24274 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.61:36161
I20260812 09:45:42.530972 24274 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/master-1/data/info.pb
I20260812 09:45:42.532559 24533 sys_catalog.cc:263] Verifying existing consensus state
I20260812 09:45:42.533373 24533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf7b481d137c42268998cd7a08f2d65e: Bootstrap starting.
I20260812 09:45:42.536597 24532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.61:36161 every 8 connection(s)
I20260812 09:45:42.542068 24533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf7b481d137c42268998cd7a08f2d65e: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 09:45:42.542402 24533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf7b481d137c42268998cd7a08f2d65e: Bootstrap complete.
I20260812 09:45:42.542959 24533 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf7b481d137c42268998cd7a08f2d65e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f8aaf46e4a6d4531ab509eb45d22e2cd" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 43093 } } peers { permanent_uuid: "cf7b481d137c42268998cd7a08f2d65e" member_type: NON_VOTER last_known_addr { host: "127.13.99.61" port: 36161 } attrs { promote: true } }
I20260812 09:45:42.543100 24533 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf7b481d137c42268998cd7a08f2d65e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cf7b481d137c42268998cd7a08f2d65e, State: Initialized, Role: LEARNER
I20260812 09:45:42.543217 24533 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf7b481d137c42268998cd7a08f2d65e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f8aaf46e4a6d4531ab509eb45d22e2cd" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 43093 } } peers { permanent_uuid: "cf7b481d137c42268998cd7a08f2d65e" member_type: NON_VOTER last_known_addr { host: "127.13.99.61" port: 36161 } attrs { promote: true } }
I20260812 09:45:42.543565 24536 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf7b481d137c42268998cd7a08f2d65e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f8aaf46e4a6d4531ab509eb45d22e2cd" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 43093 } } peers { permanent_uuid: "cf7b481d137c42268998cd7a08f2d65e" member_type: NON_VOTER last_known_addr { host: "127.13.99.61" port: 36161 } attrs { promote: true } } }
I20260812 09:45:42.543718 24536 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf7b481d137c42268998cd7a08f2d65e [sys.catalog]: This master's current role is: LEARNER
I20260812 09:45:42.543725 24533 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf7b481d137c42268998cd7a08f2d65e [sys.catalog]: configured and running, proceeding with master startup.
I20260812 09:45:42.547765 24553 catalog_manager.cc:1259] Loaded cluster ID: 5feee61a67a142d8bd855e4f84785f79
I20260812 09:45:42.547847 24553 catalog_manager.cc:1552] T 00000000000000000000000000000000 P cf7b481d137c42268998cd7a08f2d65e: loading cluster ID for follower catalog manager: success
I20260812 09:45:42.549466 24553 catalog_manager.cc:1574] T 00000000000000000000000000000000 P cf7b481d137c42268998cd7a08f2d65e: acquiring CA information for follower catalog manager: success
I20260812 09:45:42.550560 24553 catalog_manager.cc:1602] T 00000000000000000000000000000000 P cf7b481d137c42268998cd7a08f2d65e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260812 09:45:42.625771 24431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:45:42.626056 24431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:45:42.626129 24431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:45:42.630153 24431 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 09:45:42.630230 24431 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:45:42.630270 24431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:45:42.630306 24431 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 09:45:42.630337 24431 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 09:45:42.635613 24431 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:45813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.13.99.62:43093,127.13.99.61:36161,127.13.99.60:38095
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/krb5kdc/oryx_127.13.99.60.keytab
--principal=oryx/127.13.99.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.60:38095
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.13.99.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24431
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:45:42.636741 24431 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:45:42.637769 24431 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:45:42.647095 24565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:42.647090 24560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:45:42.647156 24431 server_base.cc:1034] running on GCE node
W20260812 09:45:42.647091 24562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:45:42.647852 24431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:45:42.648958 24431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:45:42.650100 24431 hybrid_clock.cc:648] HybridClock initialized: now 1786527942650073 us; error 41 us; skew 500 ppm
Aug 12 09:45:42 dist-test-slave-jq4t krb5kdc[23544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527942, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.99.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:45:42.652997 24431 init.cc:377] Logged in from keytab as oryx/127.13.99.60@KRBTEST.COM (short username oryx)
I20260812 09:45:42.653797 24431 webserver.cc:460] Webserver started at http://127.13.99.60:34825/ using document root <none> and password file <none>
I20260812 09:45:42.654310 24431 fs_manager.cc:359] Metadata directory not provided
I20260812 09:45:42.654372 24431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:45:42.654539 24431 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:45:42.656585 24431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/master-2/data/instance:
uuid: "443140f48ba94007981c7e2def497963"
format_stamp: "Formatted at 2026-08-12 09:45:42 on dist-test-slave-jq4t"
server_key: "7cdfc8f0d9482a1ad8209390a4907957"
server_key_iv: "e3e4ecd61e9c8c730b5b5ce88c5d267d"
server_key_version: "encryptionkey@0"
I20260812 09:45:42.657150 24431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/master-2/wal/instance:
uuid: "443140f48ba94007981c7e2def497963"
format_stamp: "Formatted at 2026-08-12 09:45:42 on dist-test-slave-jq4t"
server_key: "7cdfc8f0d9482a1ad8209390a4907957"
server_key_iv: "e3e4ecd61e9c8c730b5b5ce88c5d267d"
server_key_version: "encryptionkey@0"
I20260812 09:45:42.660878 24431 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260812 09:45:42.663204 24577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:42.664619 24431 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 09:45:42.664731 24431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/master-2/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/master-2/wal
uuid: "443140f48ba94007981c7e2def497963"
format_stamp: "Formatted at 2026-08-12 09:45:42 on dist-test-slave-jq4t"
server_key: "7cdfc8f0d9482a1ad8209390a4907957"
server_key_iv: "e3e4ecd61e9c8c730b5b5ce88c5d267d"
server_key_version: "encryptionkey@0"
I20260812 09:45:42.664814 24431 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:45:42.683962 24431 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:45:42.687119 24431 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:45:42.687328 24431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:45:42.695365 24431 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.60:38095
I20260812 09:45:42.695667 24646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.60:38095 every 8 connection(s)
I20260812 09:45:42.696352 24431 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/master-2/data/info.pb
I20260812 09:45:42.698598 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 24431
I20260812 09:45:42.699352 24647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:45:42.703554 24647 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.99.62" port: 43093 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 09:45:42 dist-test-slave-jq4t krb5kdc[23544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527941, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.13.99.60@KRBTEST.COM
I20260812 09:45:42.708724 23887 heartbeater.cc:503] Master 127.13.99.62:43093 was elected leader, sending a full tablet report...
Aug 12 09:45:42 dist-test-slave-jq4t krb5kdc[23544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527942, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.99.60@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM
I20260812 09:45:42.714003 24651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:42.700187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56282 (local address 127.13.99.60:38095)
0812 09:45:42.700690 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:42.700705 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:42.700741 (+ 36us) server_negotiation.cc:407] Connection header received
0812 09:45:42.701406 (+ 665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:42.701422 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:42.701742 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:42.702077 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:42.702907 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:42.703978 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:42.704898 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:42.705576 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:42.705757 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:42.708389 (+ 2632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:42.708394 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:42.708404 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:42.708430 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:42.710795 (+ 2365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:42.711303 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:42.711308 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:42.711312 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:42.711417 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:42.711964 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:42.711967 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:42.711968 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:42.712468 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:42.712802 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:42.713024 (+ 222us) server_negotiation.cc:299] Negotiation successful
0812 09:45:42.713310 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":136,"threads_started":1}
I20260812 09:45:42.714712 23658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:42.705438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54502 (local address 127.13.99.62:43093)
0812 09:45:42.705572 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:42.705576 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:42.705757 (+ 181us) server_negotiation.cc:407] Connection header received
0812 09:45:42.706014 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:42.706018 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:42.706073 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:42.706166 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:42.706822 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:42.707401 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:42.708400 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:42.708724 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:42.708878 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:42.711422 (+ 2544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:42.711426 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:42.711429 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:42.711460 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:42.712945 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:42.713479 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:42.713483 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:42.713485 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:42.713581 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:42.713867 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:42.713870 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:42.713872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:42.714052 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:42.714159 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:42.714510 (+ 351us) server_negotiation.cc:299] Negotiation successful
0812 09:45:42.714606 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260812 09:45:42.714771 24652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:42.705202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:43093 (local address 127.0.0.1:54502)
0812 09:45:42.705634 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:42.705686 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:42.705851 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:42.706218 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:42.706223 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:42.706563 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:45:42.706688 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:42.706694 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:42.707549 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:42.707552 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:42.708261 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:42.708272 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:42.708875 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:42.708879 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:42.709010 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:42.709572 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:45:42.709586 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:45:42.711284 (+ 1698us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:45:42.713060 (+ 1776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:42.713069 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:42.713075 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:42.713377 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:42.713676 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:42.713680 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:42.713682 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:42.713771 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:42.714159 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:45:42.714167 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:45:42.714290 (+ 123us) client_negotiation.cc:769] Sending connection context
0812 09:45:42.714438 (+ 148us) client_negotiation.cc:240] Negotiation successful
0812 09:45:42.714605 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":162,"threads_started":1}
I20260812 09:45:42.716019 24647 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.99.61" port: 36161 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 09:45:42 dist-test-slave-jq4t krb5kdc[23544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527942, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.99.60@KRBTEST.COM for oryx/127.13.99.61@KRBTEST.COM
I20260812 09:45:42.724311 24652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:42.716462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.61:36161 (local address 127.0.0.1:55988)
0812 09:45:42.716588 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:42.716604 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:42.716676 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:42.717072 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:42.717075 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:42.717237 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:45:42.717319 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:42.717324 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:42.718206 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:42.718211 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:42.718730 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:42.718738 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:42.719210 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:42.719214 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:42.719307 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:42.719676 (+ 369us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:45:42.719686 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:45:42.720823 (+ 1137us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:45:42.722818 (+ 1995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:42.722827 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:42.722828 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:42.723033 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:42.723291 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:42.723293 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:42.723294 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:42.723385 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:42.723837 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:45:42.723841 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:45:42.723933 (+ 92us) client_negotiation.cc:769] Sending connection context
0812 09:45:42.724040 (+ 107us) client_negotiation.cc:240] Negotiation successful
0812 09:45:42.724161 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260812 09:45:42.724363 24653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:42.716603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55988 (local address 127.13.99.61:36161)
0812 09:45:42.716863 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:42.716867 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:42.716878 (+ 11us) server_negotiation.cc:407] Connection header received
0812 09:45:42.716915 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:42.716917 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:42.716954 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:42.717052 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:42.717438 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:42.718098 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:42.718859 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:42.719095 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:42.719200 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:42.720991 (+ 1791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:42.720995 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:42.720997 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:42.721021 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:42.722675 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:42.723130 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:42.723133 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:42.723134 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:42.723211 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:42.723542 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:42.723546 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:42.723548 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:42.723732 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:42.723845 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:42.724096 (+ 251us) server_negotiation.cc:299] Negotiation successful
0812 09:45:42.724210 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":83,"threads_started":1}
I20260812 09:45:42.725122 24647 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.99.60" port: 38095 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 09:45:42 dist-test-slave-jq4t krb5kdc[23544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527942, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.99.60@KRBTEST.COM for oryx/127.13.99.60@KRBTEST.COM
I20260812 09:45:42.733191 24651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:42.725654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56292 (local address 127.13.99.60:38095)
0812 09:45:42.725750 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:42.725754 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:42.725766 (+ 12us) server_negotiation.cc:407] Connection header received
0812 09:45:42.725850 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:42.725853 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:42.725886 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:42.725974 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:42.726305 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:42.726776 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:42.728003 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:42.728232 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:42.728342 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:42.730166 (+ 1824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:42.730170 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:42.730172 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:42.730194 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:42.731574 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:42.731998 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:42.732002 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:42.732003 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:42.732077 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:42.732358 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:42.732362 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:42.732363 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:42.732595 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:42.732699 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:42.732934 (+ 235us) server_negotiation.cc:299] Negotiation successful
0812 09:45:42.733071 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":38}
I20260812 09:45:42.733175 24652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:42.725564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.60:38095 (local address 127.0.0.1:56292)
0812 09:45:42.725660 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:42.725690 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:42.725764 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:42.725985 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:42.725987 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:42.726140 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:45:42.726194 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:42.726200 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:42.727397 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:42.727400 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:42.727872 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:42.727880 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:42.728342 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:42.728345 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:42.728431 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:42.728931 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:45:42.728945 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:45:42.730041 (+ 1096us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:45:42.731701 (+ 1660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:42.731705 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:42.731707 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:42.731893 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:42.732169 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:42.732174 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:42.732176 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:42.732263 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:42.732691 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:45:42.732694 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:45:42.732777 (+ 83us) client_negotiation.cc:769] Sending connection context
0812 09:45:42.732887 (+ 110us) client_negotiation.cc:240] Negotiation successful
0812 09:45:42.733014 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260812 09:45:42.736039 24647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 443140f48ba94007981c7e2def497963: Bootstrap starting.
I20260812 09:45:42.738341 24647 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 443140f48ba94007981c7e2def497963: Neither blocks nor log segments found. Creating new log.
I20260812 09:45:42.739073 24647 log.cc:824] T 00000000000000000000000000000000 P 443140f48ba94007981c7e2def497963: Log is configured to *not* fsync() on all Append() calls
I20260812 09:45:42.740921 24647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 443140f48ba94007981c7e2def497963: No bootstrap required, opened a new log
I20260812 09:45:42.743821 24647 raft_consensus.cc:348] T 00000000000000000000000000000000 P 443140f48ba94007981c7e2def497963 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8aaf46e4a6d4531ab509eb45d22e2cd" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 43093 } } peers { permanent_uuid: "cf7b481d137c42268998cd7a08f2d65e" member_type: VOTER last_known_addr { host: "127.13.99.61" port: 36161 } } peers { permanent_uuid: "443140f48ba94007981c7e2def497963" member_type: VOTER last_known_addr { host: "127.13.99.60" port: 38095 } }
I20260812 09:45:42.744088 24647 raft_consensus.cc:374] T 00000000000000000000000000000000 P 443140f48ba94007981c7e2def497963 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:45:42.744176 24647 raft_consensus.cc:729] T 00000000000000000000000000000000 P 443140f48ba94007981c7e2def497963 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443140f48ba94007981c7e2def497963, State: Initialized, Role: FOLLOWER
I20260812 09:45:42.744717 24647 consensus_queue.cc:260] T 00000000000000000000000000000000 P 443140f48ba94007981c7e2def497963 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8aaf46e4a6d4531ab509eb45d22e2cd" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 43093 } } peers { permanent_uuid: "cf7b481d137c42268998cd7a08f2d65e" member_type: VOTER last_known_addr { host: "127.13.99.61" port: 36161 } } peers { permanent_uuid: "443140f48ba94007981c7e2def497963" member_type: VOTER last_known_addr { host: "127.13.99.60" port: 38095 } }
I20260812 09:45:42.745196 24656 sys_catalog.cc:455] T 00000000000000000000000000000000 P 443140f48ba94007981c7e2def497963 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8aaf46e4a6d4531ab509eb45d22e2cd" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 43093 } } peers { permanent_uuid: "cf7b481d137c42268998cd7a08f2d65e" member_type: VOTER last_known_addr { host: "127.13.99.61" port: 36161 } } peers { permanent_uuid: "443140f48ba94007981c7e2def497963" member_type: VOTER last_known_addr { host: "127.13.99.60" port: 38095 } } }
I20260812 09:45:42.745330 24656 sys_catalog.cc:458] T 00000000000000000000000000000000 P 443140f48ba94007981c7e2def497963 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 09:45:42.746017 24647 sys_catalog.cc:564] T 00000000000000000000000000000000 P 443140f48ba94007981c7e2def497963 [sys.catalog]: configured and running, proceeding with master startup.
W20260812 09:45:42.751055 24670 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 443140f48ba94007981c7e2def497963: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 09:45:42.751184 24670 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 09:45:42.755986 24431 master_runner.cc:428] Detected that this master 443140f48ba94007981c7e2def497963 is joining an existing cluster
I20260812 09:45:42.756052 24431 master_runner.cc:432] Initiating AddMaster RPC to add 127.13.99.60:38095
I20260812 09:45:42.767306 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 23693
I20260812 09:45:42.767592 24682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:42.758795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.60:38095 (local address 127.0.0.1:56308)
0812 09:45:42.759091 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:42.759104 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:42.759228 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:42.759516 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:42.759519 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:42.759754 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:45:42.759882 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:42.759889 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:42.760907 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:42.760910 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:42.761838 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:42.761850 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:42.762402 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:42.762405 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:42.762603 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:42.763099 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:45:42.763114 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:45:42.763562 (+ 448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:45:42.765729 (+ 2167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:42.765733 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:42.765735 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:42.765989 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:42.766434 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:42.766438 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:42.766440 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:42.766546 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:42.766965 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:45:42.766967 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:45:42.767036 (+ 69us) client_negotiation.cc:769] Sending connection context
0812 09:45:42.767159 (+ 123us) client_negotiation.cc:240] Negotiation successful
0812 09:45:42.767294 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"mutex_wait_us":35,"thread_start_us":82,"threads_started":1}
I20260812 09:45:42.767659 24651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:42.758698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56308 (local address 127.13.99.60:38095)
0812 09:45:42.758783 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:42.758787 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:42.759171 (+ 384us) server_negotiation.cc:407] Connection header received
0812 09:45:42.759318 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:42.759321 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:42.759355 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:42.759399 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:42.759999 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:42.760785 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:42.761961 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:42.762268 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:42.762478 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:42.763978 (+ 1500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:42.763983 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:42.763986 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:42.764008 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:42.765618 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:42.766092 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:42.766095 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:42.766096 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:42.766170 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:42.766650 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:42.766653 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:42.766655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:42.766845 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:42.766910 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:42.767261 (+ 351us) server_negotiation.cc:299] Negotiation successful
0812 09:45:42.767387 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":147,"server-negotiator.queue_time_us":28,"spinlock_wait_cycles":3328}
I20260812 09:45:42.784859 24679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:42.758127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:43093 (local address 127.0.0.1:54518)
0812 09:45:42.758501 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:42.758518 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:42.758611 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:42.759027 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:42.759030 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:42.759268 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:45:42.759414 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:42.759420 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:42.760543 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:42.760546 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:42.762351 (+ 1805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:42.762361 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:42.763398 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:42.763402 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:42.763500 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:42.763994 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:45:42.764027 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:45:42.780013 (+ 15986us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:45:42.783088 (+ 3075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:42.783092 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:42.783094 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:42.783339 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:42.783663 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:42.783667 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:42.783669 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:42.783759 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:42.784234 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:45:42.784237 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:45:42.784356 (+ 119us) client_negotiation.cc:769] Sending connection context
0812 09:45:42.784459 (+ 103us) client_negotiation.cc:240] Negotiation successful
0812 09:45:42.784648 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"mutex_wait_us":4,"spinlock_wait_cycles":8192,"thread_start_us":99,"threads_started":1}
I20260812 09:45:42.785549 24653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:42.758304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55996 (local address 127.13.99.61:36161)
0812 09:45:42.758444 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:42.758448 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:42.758966 (+ 518us) server_negotiation.cc:407] Connection header received
0812 09:45:42.759152 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:42.759156 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:42.759208 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:42.759260 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:42.759886 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:42.760641 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:42.762099 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:42.762484 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:42.762559 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:42.781391 (+ 18832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:42.781399 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:42.781405 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:42.781443 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:42.783306 (+ 1863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:42.783882 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:42.783889 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:42.783893 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:42.784000 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:42.784370 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:42.784376 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:42.784381 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:42.784675 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:42.784755 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:42.785144 (+ 389us) server_negotiation.cc:299] Negotiation successful
0812 09:45:42.785302 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260812 09:45:42.785560 23658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:42.758578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54518 (local address 127.13.99.62:43093)
0812 09:45:42.758705 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:42.758710 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:42.758724 (+ 14us) server_negotiation.cc:407] Connection header received
0812 09:45:42.758763 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:42.758766 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:42.758814 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:42.758888 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:42.759566 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:42.760391 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:42.762527 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:42.763267 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:42.763398 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:42.780246 (+ 16848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:42.780251 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:42.780254 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:42.780637 (+ 383us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:42.782949 (+ 2312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:42.783454 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:42.783458 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:42.783460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:42.783557 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:42.783859 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:42.783863 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:42.783864 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:42.784095 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:42.784189 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:42.785262 (+ 1073us) server_negotiation.cc:299] Negotiation successful
0812 09:45:42.785390 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260812 09:45:42.786182 24681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:42.758208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.61:36161 (local address 127.0.0.1:55996)
0812 09:45:42.758893 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:42.758905 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:42.759036 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:42.759407 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:42.759410 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:42.759646 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:45:42.759751 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:42.759758 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:42.760780 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:42.760786 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:42.761949 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:42.761961 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:42.762705 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:42.762709 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:42.762830 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:42.780654 (+ 17824us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:45:42.780676 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:45:42.781169 (+ 493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:45:42.783448 (+ 2279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:42.783454 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:42.783459 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:42.783749 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:42.784142 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:42.784148 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:42.784153 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:42.784257 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:42.784834 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:45:42.784840 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:45:42.784949 (+ 109us) client_negotiation.cc:769] Sending connection context
0812 09:45:42.785782 (+ 833us) client_negotiation.cc:240] Negotiation successful
0812 09:45:42.785960 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"mutex_wait_us":74,"thread_start_us":108,"threads_started":1}
I20260812 09:45:42.788610 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.99.1:33935
--local_ip_for_outbound_sockets=127.13.99.1
--tserver_master_addrs=127.13.99.62:43093
--webserver_port=46441
--webserver_interface=127.13.99.1
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/krb5kdc/oryx_127.13.99.1.keytab
--principal=oryx/127.13.99.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:45813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.13.99.62:43093,127.13.99.61:36161
--tserver_master_addrs=127.13.99.62:43093,127.13.99.61:36161,127.13.99.60:38095 with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 09:45:42 dist-test-slave-jq4t krb5kdc[23544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527941, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.99.62@KRBTEST.COM for oryx/127.13.99.60@KRBTEST.COM
I20260812 09:45:42.800650 24651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:42.790835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56322 (local address 127.13.99.60:38095)
0812 09:45:42.790970 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:42.790974 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:42.790987 (+ 13us) server_negotiation.cc:407] Connection header received
0812 09:45:42.791056 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:42.791059 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:42.791131 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:42.791208 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:42.791757 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:42.792660 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:42.793848 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:42.794269 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:42.794398 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:42.796922 (+ 2524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:42.796926 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:42.796929 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:42.796955 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:42.798675 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:42.799237 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:42.799240 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:42.799242 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:42.799328 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:42.799669 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:42.799672 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:42.799674 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:42.799873 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:42.799969 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:42.800346 (+ 377us) server_negotiation.cc:299] Negotiation successful
0812 09:45:42.800457 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260812 09:45:42.800750 24417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:42.790222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.60:38095 (local address 127.0.0.1:56322)
0812 09:45:42.790537 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:42.790557 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:42.790645 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:42.791327 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:42.791330 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:42.791515 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:45:42.791627 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:42.791634 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:42.792797 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:42.792801 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:42.793703 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:42.793720 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:42.794409 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:42.794413 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:42.794514 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:42.795022 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:45:42.795037 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:45:42.796771 (+ 1734us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:45:42.798798 (+ 2027us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:42.798804 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:42.798809 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:42.799139 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:42.799437 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:42.799442 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:42.799446 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:42.799563 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:42.799987 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:45:42.800019 (+ 32us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:45:42.800146 (+ 127us) client_negotiation.cc:769] Sending connection context
0812 09:45:42.800282 (+ 136us) client_negotiation.cc:240] Negotiation successful
0812 09:45:42.800454 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218}
I20260812 09:45:42.802098 23606 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.13.99.60:38095: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "443140f48ba94007981c7e2def497963"
member_type: NON_VOTER
last_known_addr {
host: "127.13.99.60"
port: 38095
}
attrs {
promote: true
}
}
dest_uuid: "f8aaf46e4a6d4531ab509eb45d22e2cd"
cas_config_opid_index: 5
I20260812 09:45:42.802764 23606 consensus_queue.cc:237] T 00000000000000000000000000000000 P f8aaf46e4a6d4531ab509eb45d22e2cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f8aaf46e4a6d4531ab509eb45d22e2cd" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 43093 } } peers { permanent_uuid: "cf7b481d137c42268998cd7a08f2d65e" member_type: NON_VOTER last_known_addr { host: "127.13.99.61" port: 36161 } attrs { promote: true } } peers { permanent_uuid: "443140f48ba94007981c7e2def497963" member_type: NON_VOTER last_known_addr { host: "127.13.99.60" port: 38095 } attrs { promote: true } }
I20260812 09:45:42.803807 24420 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f8aaf46e4a6d4531ab509eb45d22e2cd [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 443140f48ba94007981c7e2def497963 (127.13.99.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f8aaf46e4a6d4531ab509eb45d22e2cd" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 43093 } } peers { permanent_uuid: "cf7b481d137c42268998cd7a08f2d65e" member_type: NON_VOTER last_known_addr { host: "127.13.99.61" port: 36161 } attrs { promote: true } } peers { permanent_uuid: "443140f48ba94007981c7e2def497963" member_type: NON_VOTER last_known_addr { host: "127.13.99.60" port: 38095 } attrs { promote: true } } }
I20260812 09:45:42.805330 24420 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.13.99.60:38095
I20260812 09:45:42.805503 24420 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8aaf46e4a6d4531ab509eb45d22e2cd [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f8aaf46e4a6d4531ab509eb45d22e2cd" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f8aaf46e4a6d4531ab509eb45d22e2cd" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 43093 } } peers { permanent_uuid: "cf7b481d137c42268998cd7a08f2d65e" member_type: NON_VOTER last_known_addr { host: "127.13.99.61" port: 36161 } attrs { promote: true } } peers { permanent_uuid: "443140f48ba94007981c7e2def497963" member_type: NON_VOTER last_known_addr { host: "127.13.99.60" port: 38095 } attrs { promote: true } } }
I20260812 09:45:42.805606 24420 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8aaf46e4a6d4531ab509eb45d22e2cd [sys.catalog]: This master's current role is: LEADER
I20260812 09:45:42.806295 24611 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 443140f48ba94007981c7e2def497963 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:45:42.807721 24611 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 443140f48ba94007981c7e2def497963 [term 1 FOLLOWER]: Refusing update from remote peer f8aaf46e4a6d4531ab509eb45d22e2cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260812 09:45:42.808405 24431 master.cc:537] Master@127.13.99.60:38095 shutting down...
I20260812 09:45:42.808612 24421 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f8aaf46e4a6d4531ab509eb45d22e2cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "443140f48ba94007981c7e2def497963" member_type: NON_VOTER last_known_addr { host: "127.13.99.60" port: 38095 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 09:45:42.810659 23588 consensus_peers.cc:487] T 00000000000000000000000000000000 P f8aaf46e4a6d4531ab509eb45d22e2cd -> Peer 443140f48ba94007981c7e2def497963 (127.13.99.60:38095): Couldn't send request to peer 443140f48ba94007981c7e2def497963. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260812 09:45:42.813954 24431 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 443140f48ba94007981c7e2def497963 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 09:45:42.814298 24431 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 443140f48ba94007981c7e2def497963 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 09:45:42.814465 24431 tablet_replica.cc:330] T 00000000000000000000000000000000 P 443140f48ba94007981c7e2def497963: stopping tablet replica
I20260812 09:45:42.816272 24653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:42.805027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56010 (local address 127.13.99.61:36161)
0812 09:45:42.805160 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:42.805167 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:42.810252 (+ 5085us) server_negotiation.cc:407] Connection header received
0812 09:45:42.810461 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:42.810465 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:42.810536 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:42.810613 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:45:42.811300 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:42.812120 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:42.814723 (+ 2603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:42.815427 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:42.815512 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:42.815676 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:42.816065 (+ 389us) server_negotiation.cc:299] Negotiation successful
0812 09:45:42.816142 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260812 09:45:42.816987 24417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:42.804925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.61:36161 (local address 127.0.0.1:56010)
0812 09:45:42.810173 (+ 5248us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:42.810187 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:42.810330 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:42.810785 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:42.810788 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:42.811072 (+ 284us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:45:42.811164 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:42.811172 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:42.812675 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:42.812678 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:42.814498 (+ 1820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:42.814510 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:42.815778 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:42.815783 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:42.815887 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:42.815932 (+ 45us) client_negotiation.cc:769] Sending connection context
0812 09:45:42.816459 (+ 527us) client_negotiation.cc:240] Negotiation successful
0812 09:45:42.816863 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5183}
I20260812 09:45:42.817508 24492 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cf7b481d137c42268998cd7a08f2d65e [term 1 LEARNER]: Refusing update from remote peer f8aaf46e4a6d4531ab509eb45d22e2cd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260812 09:45:42.818060 24421 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f8aaf46e4a6d4531ab509eb45d22e2cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf7b481d137c42268998cd7a08f2d65e" member_type: NON_VOTER last_known_addr { host: "127.13.99.61" port: 36161 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260812 09:45:42.819866 24431 master.cc:552] Master@127.13.99.60:38095 shutdown complete.
I20260812 09:45:42.819955 24431 master_runner.cc:305] Clearing existing system tablet
I20260812 09:45:42.820233 24431 fs_manager.cc:359] Metadata directory not provided
I20260812 09:45:42.820325 24431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:45:42.821329 24492 raft_consensus.cc:2932] T 00000000000000000000000000000000 P cf7b481d137c42268998cd7a08f2d65e [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 443140f48ba94007981c7e2def497963 (127.13.99.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f8aaf46e4a6d4531ab509eb45d22e2cd" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 43093 } } peers { permanent_uuid: "cf7b481d137c42268998cd7a08f2d65e" member_type: NON_VOTER last_known_addr { host: "127.13.99.61" port: 36161 } attrs { promote: true } } peers { permanent_uuid: "443140f48ba94007981c7e2def497963" member_type: NON_VOTER last_known_addr { host: "127.13.99.60" port: 38095 } attrs { promote: true } } }
I20260812 09:45:42.823396 24536 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf7b481d137c42268998cd7a08f2d65e [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8aaf46e4a6d4531ab509eb45d22e2cd. Latest consensus state: current_term: 1 leader_uuid: "f8aaf46e4a6d4531ab509eb45d22e2cd" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f8aaf46e4a6d4531ab509eb45d22e2cd" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 43093 } } peers { permanent_uuid: "cf7b481d137c42268998cd7a08f2d65e" member_type: NON_VOTER last_known_addr { host: "127.13.99.61" port: 36161 } attrs { promote: true } } peers { permanent_uuid: "443140f48ba94007981c7e2def497963" member_type: NON_VOTER last_known_addr { host: "127.13.99.60" port: 38095 } attrs { promote: true } } }
I20260812 09:45:42.823521 24536 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf7b481d137c42268998cd7a08f2d65e [sys.catalog]: This master's current role is: LEARNER
I20260812 09:45:42.823426 24690 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf7b481d137c42268998cd7a08f2d65e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f8aaf46e4a6d4531ab509eb45d22e2cd" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f8aaf46e4a6d4531ab509eb45d22e2cd" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 43093 } } peers { permanent_uuid: "cf7b481d137c42268998cd7a08f2d65e" member_type: NON_VOTER last_known_addr { host: "127.13.99.61" port: 36161 } attrs { promote: true } } peers { permanent_uuid: "443140f48ba94007981c7e2def497963" member_type: NON_VOTER last_known_addr { host: "127.13.99.60" port: 38095 } attrs { promote: true } } }
I20260812 09:45:42.823698 24690 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf7b481d137c42268998cd7a08f2d65e [sys.catalog]: This master's current role is: LEARNER
I20260812 09:45:42.824807 24431 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.000s
I20260812 09:45:42.824837 24421 raft_consensus.cc:1050] T 00000000000000000000000000000000 P f8aaf46e4a6d4531ab509eb45d22e2cd: attempting to promote NON_VOTER cf7b481d137c42268998cd7a08f2d65e to VOTER
I20260812 09:45:42.825260 24421 consensus_queue.cc:237] T 00000000000000000000000000000000 P f8aaf46e4a6d4531ab509eb45d22e2cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f8aaf46e4a6d4531ab509eb45d22e2cd" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 43093 } } peers { permanent_uuid: "cf7b481d137c42268998cd7a08f2d65e" member_type: VOTER last_known_addr { host: "127.13.99.61" port: 36161 } attrs { promote: false } } peers { permanent_uuid: "443140f48ba94007981c7e2def497963" member_type: NON_VOTER last_known_addr { host: "127.13.99.60" port: 38095 } attrs { promote: true } }
I20260812 09:45:42.826864 24492 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cf7b481d137c42268998cd7a08f2d65e [term 1 LEARNER]: Refusing update from remote peer f8aaf46e4a6d4531ab509eb45d22e2cd: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260812 09:45:42.827312 24420 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f8aaf46e4a6d4531ab509eb45d22e2cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf7b481d137c42268998cd7a08f2d65e" member_type: VOTER last_known_addr { host: "127.13.99.61" port: 36161 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260812 09:45:42.828557 24417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:42.826339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.60:38095 (local address 127.0.0.1:56338)
0812 09:45:42.828378 (+ 2039us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:42.828460 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.60:38095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1948}
I20260812 09:45:42.828830 24693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:42.829068 24431 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:45:42.829181 24431 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260812 09:45:42.829213 24431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/master-2/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/master-2/wal
uuid: "443140f48ba94007981c7e2def497963"
format_stamp: "Formatted at 2026-08-12 09:45:42 on dist-test-slave-jq4t"
server_key: "7cdfc8f0d9482a1ad8209390a4907957"
server_key_iv: "e3e4ecd61e9c8c730b5b5ce88c5d267d"
server_key_version: "encryptionkey@0"
I20260812 09:45:42.829282 24417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:42.829052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.60:38095 (local address 127.0.0.1:56350)
0812 09:45:42.829180 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:42.829224 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.60:38095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260812 09:45:42.829407 23588 consensus_peers.cc:487] T 00000000000000000000000000000000 P f8aaf46e4a6d4531ab509eb45d22e2cd -> Peer 443140f48ba94007981c7e2def497963 (127.13.99.60:38095): Couldn't send request to peer 443140f48ba94007981c7e2def497963. Status: Network error: Client connection negotiation failed: client connection to 127.13.99.60:38095: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260812 09:45:42.831370 24431 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 443140f48ba94007981c7e2def497963: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 09:45:42.833905 24431 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 443140f48ba94007981c7e2def497963: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260812 09:45:42.833985 24431 log.cc:1196] T 00000000000000000000000000000000 P 443140f48ba94007981c7e2def497963: Deleting WAL directory at /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260812 09:45:42.834227 24431 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 443140f48ba94007981c7e2def497963: Deleting consensus metadata
I20260812 09:45:42.834378 24431 master_runner.cc:315] Copying system tablet from 127.13.99.62:43093
I20260812 09:45:42.835031 24420 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f8aaf46e4a6d4531ab509eb45d22e2cd [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, cf7b481d137c42268998cd7a08f2d65e (127.13.99.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f8aaf46e4a6d4531ab509eb45d22e2cd" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 43093 } } peers { permanent_uuid: "cf7b481d137c42268998cd7a08f2d65e" member_type: VOTER last_known_addr { host: "127.13.99.61" port: 36161 } attrs { promote: false } } peers { permanent_uuid: "443140f48ba94007981c7e2def497963" member_type: NON_VOTER last_known_addr { host: "127.13.99.60" port: 38095 } attrs { promote: true } } }
I20260812 09:45:42.835539 24431 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 443140f48ba94007981c7e2def497963: tablet copy: Beginning tablet copy session from remote peer at address 127.13.99.62:43093
I20260812 09:45:42.836705 24420 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8aaf46e4a6d4531ab509eb45d22e2cd [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f8aaf46e4a6d4531ab509eb45d22e2cd" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f8aaf46e4a6d4531ab509eb45d22e2cd" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 43093 } } peers { permanent_uuid: "cf7b481d137c42268998cd7a08f2d65e" member_type: VOTER last_known_addr { host: "127.13.99.61" port: 36161 } attrs { promote: false } } peers { permanent_uuid: "443140f48ba94007981c7e2def497963" member_type: NON_VOTER last_known_addr { host: "127.13.99.60" port: 38095 } attrs { promote: true } } }
I20260812 09:45:42.836808 24420 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8aaf46e4a6d4531ab509eb45d22e2cd [sys.catalog]: This master's current role is: LEADER
I20260812 09:45:42.837160 24420 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8aaf46e4a6d4531ab509eb45d22e2cd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f8aaf46e4a6d4531ab509eb45d22e2cd" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f8aaf46e4a6d4531ab509eb45d22e2cd" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 43093 } } peers { permanent_uuid: "cf7b481d137c42268998cd7a08f2d65e" member_type: VOTER last_known_addr { host: "127.13.99.61" port: 36161 } attrs { promote: false } } peers { permanent_uuid: "443140f48ba94007981c7e2def497963" member_type: NON_VOTER last_known_addr { host: "127.13.99.60" port: 38095 } attrs { promote: true } } }
I20260812 09:45:42.837263 24420 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8aaf46e4a6d4531ab509eb45d22e2cd [sys.catalog]: This master's current role is: LEADER
I20260812 09:45:42.837968 24492 raft_consensus.cc:2932] T 00000000000000000000000000000000 P cf7b481d137c42268998cd7a08f2d65e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, cf7b481d137c42268998cd7a08f2d65e (127.13.99.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f8aaf46e4a6d4531ab509eb45d22e2cd" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 43093 } } peers { permanent_uuid: "cf7b481d137c42268998cd7a08f2d65e" member_type: VOTER last_known_addr { host: "127.13.99.61" port: 36161 } attrs { promote: false } } peers { permanent_uuid: "443140f48ba94007981c7e2def497963" member_type: NON_VOTER last_known_addr { host: "127.13.99.60" port: 38095 } attrs { promote: true } } }
I20260812 09:45:42.839522 24690 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf7b481d137c42268998cd7a08f2d65e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f8aaf46e4a6d4531ab509eb45d22e2cd" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f8aaf46e4a6d4531ab509eb45d22e2cd" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 43093 } } peers { permanent_uuid: "cf7b481d137c42268998cd7a08f2d65e" member_type: VOTER last_known_addr { host: "127.13.99.61" port: 36161 } attrs { promote: false } } peers { permanent_uuid: "443140f48ba94007981c7e2def497963" member_type: NON_VOTER last_known_addr { host: "127.13.99.60" port: 38095 } attrs { promote: true } } }
I20260812 09:45:42.839628 24690 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf7b481d137c42268998cd7a08f2d65e [sys.catalog]: This master's current role is: FOLLOWER
I20260812 09:45:42.847340 23658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:42.837953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54534 (local address 127.13.99.62:43093)
0812 09:45:42.838062 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:42.838067 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:42.838499 (+ 432us) server_negotiation.cc:407] Connection header received
0812 09:45:42.838713 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:42.838719 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:42.838775 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:42.838835 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:42.839693 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:42.840521 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:42.841801 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:42.842149 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:42.842256 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:42.843684 (+ 1428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:42.843688 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:42.843695 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:42.843723 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:42.845522 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:42.846001 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:42.846005 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:42.846006 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:42.846100 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:42.846393 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:42.846396 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:42.846398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:42.846617 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:42.846714 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:42.847074 (+ 360us) server_negotiation.cc:299] Negotiation successful
0812 09:45:42.847190 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260812 09:45:42.847856 24705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:42.838161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:43093 (local address 127.0.0.1:54534)
0812 09:45:42.838416 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:42.838433 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:42.838590 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:42.838994 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:42.838997 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:42.839237 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:45:42.839554 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:42.839562 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:42.840776 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:42.840779 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:42.841669 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:42.841680 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:42.842386 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:42.842390 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:42.842490 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:42.843070 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:45:42.843085 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:45:42.843538 (+ 453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:45:42.845637 (+ 2099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:42.845641 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:42.845644 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:42.845900 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:42.846202 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:42.846205 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:42.846206 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:42.846299 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:42.846739 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:45:42.846742 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:45:42.846863 (+ 121us) client_negotiation.cc:769] Sending connection context
0812 09:45:42.847006 (+ 143us) client_negotiation.cc:240] Negotiation successful
0812 09:45:42.847206 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":66,"threads_started":1}
I20260812 09:45:42.848589 23633 tablet_copy_service.cc:140] P f8aaf46e4a6d4531ab509eb45d22e2cd: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 443140f48ba94007981c7e2def497963 ({username='oryx', principal='oryx/127.13.99.60@KRBTEST.COM'} at 127.0.0.1:54534)
I20260812 09:45:42.848690 23633 tablet_copy_service.cc:161] P f8aaf46e4a6d4531ab509eb45d22e2cd: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 443140f48ba94007981c7e2def497963 at {username='oryx', principal='oryx/127.13.99.60@KRBTEST.COM'} at 127.0.0.1:54534: session id = 443140f48ba94007981c7e2def497963-00000000000000000000000000000000
I20260812 09:45:42.849728 23633 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P f8aaf46e4a6d4531ab509eb45d22e2cd: Tablet Copy: opened 0 blocks and 1 log segments
I20260812 09:45:42.851090 24431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260812 09:45:42.853471 24431 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 443140f48ba94007981c7e2def497963: tablet copy: Starting download of 0 data blocks...
I20260812 09:45:42.853663 24431 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 443140f48ba94007981c7e2def497963: tablet copy: Starting download of 1 WAL segments...
I20260812 09:45:42.855208 24431 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 443140f48ba94007981c7e2def497963: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260812 09:45:42.857018 23633 tablet_copy_service.cc:342] P f8aaf46e4a6d4531ab509eb45d22e2cd: Request end of tablet copy session 443140f48ba94007981c7e2def497963-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.13.99.60@KRBTEST.COM'} at 127.0.0.1:54534
I20260812 09:45:42.857096 23633 tablet_copy_service.cc:433] P f8aaf46e4a6d4531ab509eb45d22e2cd: ending tablet copy session 443140f48ba94007981c7e2def497963-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 443140f48ba94007981c7e2def497963
I20260812 09:45:42.864296 24431 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:45:42.866012 24708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:42.866194 24712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:45:42.866350 24431 server_base.cc:1034] running on GCE node
W20260812 09:45:42.866609 24710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:45:42.866827 24431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:45:42.867100 24431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:45:42.868249 24431 hybrid_clock.cc:648] HybridClock initialized: now 1786527942868236 us; error 29 us; skew 500 ppm
Aug 12 09:45:42 dist-test-slave-jq4t krb5kdc[23544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527942, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.99.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:45:42.869916 24431 init.cc:377] Logged in from keytab as oryx/127.13.99.60@KRBTEST.COM (short username oryx)
I20260812 09:45:42.870375 24431 webserver.cc:460] Webserver started at http://127.13.99.60:40177/ using document root <none> and password file <none>
I20260812 09:45:42.870573 24431 fs_manager.cc:359] Metadata directory not provided
I20260812 09:45:42.870622 24431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:45:42.871950 24431 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 09:45:42.872998 24726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:42.873286 24431 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260812 09:45:42.873358 24431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/master-2/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/master-2/wal
uuid: "443140f48ba94007981c7e2def497963"
format_stamp: "Formatted at 2026-08-12 09:45:42 on dist-test-slave-jq4t"
server_key: "7cdfc8f0d9482a1ad8209390a4907957"
server_key_iv: "e3e4ecd61e9c8c730b5b5ce88c5d267d"
server_key_version: "encryptionkey@0"
I20260812 09:45:42.873427 24431 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:45:42.899850 24431 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:45:42.902225 24431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:45:42.907498 24431 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.60:38095
I20260812 09:45:42.908107 24431 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/master-2/data/info.pb
I20260812 09:45:42.909626 24802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.60:38095 every 8 connection(s)
I20260812 09:45:42.911306 24803 sys_catalog.cc:263] Verifying existing consensus state
I20260812 09:45:42.912644 24803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 443140f48ba94007981c7e2def497963: Bootstrap starting.
I20260812 09:45:42.924254 24803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 443140f48ba94007981c7e2def497963: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 09:45:42.924582 24803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 443140f48ba94007981c7e2def497963: Bootstrap complete.
I20260812 09:45:42.925163 24803 raft_consensus.cc:348] T 00000000000000000000000000000000 P 443140f48ba94007981c7e2def497963 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f8aaf46e4a6d4531ab509eb45d22e2cd" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 43093 } } peers { permanent_uuid: "cf7b481d137c42268998cd7a08f2d65e" member_type: VOTER last_known_addr { host: "127.13.99.61" port: 36161 } attrs { promote: false } } peers { permanent_uuid: "443140f48ba94007981c7e2def497963" member_type: NON_VOTER last_known_addr { host: "127.13.99.60" port: 38095 } attrs { promote: true } }
I20260812 09:45:42.925309 24803 raft_consensus.cc:729] T 00000000000000000000000000000000 P 443140f48ba94007981c7e2def497963 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 443140f48ba94007981c7e2def497963, State: Initialized, Role: LEARNER
I20260812 09:45:42.925419 24803 consensus_queue.cc:260] T 00000000000000000000000000000000 P 443140f48ba94007981c7e2def497963 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f8aaf46e4a6d4531ab509eb45d22e2cd" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 43093 } } peers { permanent_uuid: "cf7b481d137c42268998cd7a08f2d65e" member_type: VOTER last_known_addr { host: "127.13.99.61" port: 36161 } attrs { promote: false } } peers { permanent_uuid: "443140f48ba94007981c7e2def497963" member_type: NON_VOTER last_known_addr { host: "127.13.99.60" port: 38095 } attrs { promote: true } }
I20260812 09:45:42.925781 24807 sys_catalog.cc:455] T 00000000000000000000000000000000 P 443140f48ba94007981c7e2def497963 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f8aaf46e4a6d4531ab509eb45d22e2cd" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 43093 } } peers { permanent_uuid: "cf7b481d137c42268998cd7a08f2d65e" member_type: VOTER last_known_addr { host: "127.13.99.61" port: 36161 } attrs { promote: false } } peers { permanent_uuid: "443140f48ba94007981c7e2def497963" member_type: NON_VOTER last_known_addr { host: "127.13.99.60" port: 38095 } attrs { promote: true } } }
I20260812 09:45:42.925892 24803 sys_catalog.cc:564] T 00000000000000000000000000000000 P 443140f48ba94007981c7e2def497963 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 09:45:42.926018 24807 sys_catalog.cc:458] T 00000000000000000000000000000000 P 443140f48ba94007981c7e2def497963 [sys.catalog]: This master's current role is: LEARNER
I20260812 09:45:42.926558 24081 heartbeater.cc:503] Master 127.13.99.62:43093 was elected leader, sending a full tablet report...
I20260812 09:45:42.929450 24824 catalog_manager.cc:1259] Loaded cluster ID: 5feee61a67a142d8bd855e4f84785f79
I20260812 09:45:42.929518 24824 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 443140f48ba94007981c7e2def497963: loading cluster ID for follower catalog manager: success
I20260812 09:45:42.931757 24824 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 443140f48ba94007981c7e2def497963: acquiring CA information for follower catalog manager: success
I20260812 09:45:42.933364 24824 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 443140f48ba94007981c7e2def497963: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260812 09:45:42.938742 24683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:45:42.938974 24683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:45:42.939060 24683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:45:42.943189 24683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:45:42.943259 24683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:45:42.943358 24683 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.99.1
I20260812 09:45:42.948781 24683 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:45813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/krb5kdc/oryx_127.13.99.1.keytab
--principal=oryx/127.13.99.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.99.1:33935
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.99.1
--webserver_port=46441
--tserver_master_addrs=127.13.99.62:43093,127.13.99.61:36161,127.13.99.60:38095
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24683
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.99.1
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:45:42.949827 24683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:45:42.950865 24683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:45:42.960554 24834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:42.960520 24831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:42.960580 24829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:45:42.960777 24683 server_base.cc:1034] running on GCE node
I20260812 09:45:42.961182 24683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:45:42.961799 24683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:45:42.962996 24683 hybrid_clock.cc:648] HybridClock initialized: now 1786527942962984 us; error 23 us; skew 500 ppm
Aug 12 09:45:42 dist-test-slave-jq4t krb5kdc[23544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527942, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.99.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:45:42.965746 24683 init.cc:377] Logged in from keytab as oryx/127.13.99.1@KRBTEST.COM (short username oryx)
I20260812 09:45:42.966571 24683 webserver.cc:460] Webserver started at http://127.13.99.1:46441/ using document root <none> and password file <none>
I20260812 09:45:42.967129 24683 fs_manager.cc:359] Metadata directory not provided
I20260812 09:45:42.967247 24683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:45:42.971071 24683 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260812 09:45:42.973199 24846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:42.974406 24683 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 09:45:42.974558 24683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/ts-0/wal
uuid: "22cbcc6238b04ef196d61a4623d51639"
format_stamp: "Formatted at 2026-08-12 09:45:41 on dist-test-slave-jq4t"
server_key: "a494b14262578f0cfd435f5db85c1a63"
server_key_iv: "bfc70db1382ee4ab680bfc03b914029d"
server_key_version: "encryptionkey@0"
I20260812 09:45:42.975095 24683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:45:42.984999 24683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:45:42.987622 24683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:45:42.987818 24683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:45:42.988387 24683 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 09:45:42.989246 24683 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:45:42.989318 24683 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:42.989387 24683 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:45:42.989441 24683 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:42.999657 24683 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.1:33935
I20260812 09:45:42.999732 25020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.1:33935 every 8 connection(s)
I20260812 09:45:43.000804 24683 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/ts-0/data/info.pb
I20260812 09:45:43.009480 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 24683
I20260812 09:45:43.009688 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 23893
Aug 12 09:45:43 dist-test-slave-jq4t krb5kdc[23544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527942, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.99.1@KRBTEST.COM for oryx/127.13.99.61@KRBTEST.COM
Aug 12 09:45:43 dist-test-slave-jq4t krb5kdc[23544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527942, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.99.1@KRBTEST.COM for oryx/127.13.99.60@KRBTEST.COM
Aug 12 09:45:43 dist-test-slave-jq4t krb5kdc[23544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527942, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.99.1@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM
I20260812 09:45:43.021139 25032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:43.004621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.1:54679 (local address 127.13.99.60:38095)
0812 09:45:43.004847 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:43.004850 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:43.004868 (+ 18us) server_negotiation.cc:407] Connection header received
0812 09:45:43.005872 (+ 1004us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:43.005875 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:43.005929 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:43.006032 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:43.007073 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:43.007785 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:43.009448 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:43.009849 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:43.009943 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:43.016171 (+ 6228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:43.016177 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:43.016184 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:43.016215 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:43.018118 (+ 1903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:43.018803 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:43.018809 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:43.018811 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:43.018906 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:43.019325 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:43.019328 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:43.019330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:43.019581 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:43.019677 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:43.020827 (+ 1150us) server_negotiation.cc:299] Negotiation successful
0812 09:45:43.020960 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":83,"threads_started":1}
I20260812 09:45:43.021196 24653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:43.002770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.1:39421 (local address 127.13.99.61:36161)
0812 09:45:43.002938 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:43.002942 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:43.004143 (+ 1201us) server_negotiation.cc:407] Connection header received
0812 09:45:43.005451 (+ 1308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:43.005455 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:43.005523 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:43.005633 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:43.007088 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:43.007874 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:43.009425 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:43.009781 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:43.009879 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:43.016226 (+ 6347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:43.016234 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:43.016240 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:43.016285 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:43.018444 (+ 2159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:43.019117 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:43.019124 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:43.019129 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:43.019246 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:43.019719 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:43.019726 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:43.019731 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:43.020047 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:43.020159 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:43.020707 (+ 548us) server_negotiation.cc:299] Negotiation successful
0812 09:45:43.020906 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260812 09:45:43.021831 25028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:43.003317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:43093 (local address 127.13.99.1:38461)
0812 09:45:43.003959 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:43.004023 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:43.005210 (+ 1187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:43.005815 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:43.005820 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:43.006338 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:45:43.006878 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:43.006898 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:43.007977 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:43.007982 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:43.009300 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:43.009307 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:43.010111 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:43.010114 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:43.010451 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:43.014666 (+ 4215us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:45:43.014677 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:45:43.015882 (+ 1205us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:45:43.018873 (+ 2991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:43.018876 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:43.018878 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:43.019116 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:43.019440 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:43.019443 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:43.019445 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:43.019581 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:43.020041 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:45:43.020044 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:45:43.020118 (+ 74us) client_negotiation.cc:769] Sending connection context
0812 09:45:43.020287 (+ 169us) client_negotiation.cc:240] Negotiation successful
0812 09:45:43.020729 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"mutex_wait_us":320,"spinlock_wait_cycles":72320,"thread_start_us":157,"threads_started":1}
I20260812 09:45:43.021831 25031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:43.004481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.60:38095 (local address 127.13.99.1:54679)
0812 09:45:43.004797 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:43.004812 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:43.005221 (+ 409us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:43.006073 (+ 852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:43.006077 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:43.006405 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:45:43.006891 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:43.006901 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:43.007935 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:43.007940 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:43.009295 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:43.009307 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:43.010107 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:43.010111 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:43.012925 (+ 2814us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:43.013430 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:45:43.013448 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:45:43.016000 (+ 2552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:45:43.018271 (+ 2271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:43.018280 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:43.018300 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:43.018675 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:43.019025 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:43.019029 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:43.019031 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:43.019218 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:43.019729 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:45:43.019739 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:45:43.020083 (+ 344us) client_negotiation.cc:769] Sending connection context
0812 09:45:43.020287 (+ 204us) client_negotiation.cc:240] Negotiation successful
0812 09:45:43.021600 (+ 1313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"mutex_wait_us":3959,"spinlock_wait_cycles":19584,"thread_start_us":96,"threads_started":1}
I20260812 09:45:43.022732 25030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:43.003317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.61:36161 (local address 127.13.99.1:39421)
0812 09:45:43.003959 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:43.004023 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:43.005210 (+ 1187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:43.005811 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:43.005819 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:43.006392 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:45:43.006894 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:43.006901 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:43.008028 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:43.008032 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:43.009248 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:43.009264 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:43.010106 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:43.010113 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:43.010462 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:43.011596 (+ 1134us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:45:43.011625 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:45:43.015460 (+ 3835us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:45:43.018608 (+ 3148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:43.018615 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:43.018620 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:43.018965 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:43.019387 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:43.019405 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:43.019410 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:43.019574 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:43.020238 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:45:43.020244 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:45:43.020379 (+ 135us) client_negotiation.cc:769] Sending connection context
0812 09:45:43.021536 (+ 1157us) client_negotiation.cc:240] Negotiation successful
0812 09:45:43.021743 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"mutex_wait_us":268,"spinlock_wait_cycles":5760,"thread_start_us":165,"threads_started":1}
I20260812 09:45:43.022753 23658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:43.002943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.1:38461 (local address 127.13.99.62:43093)
0812 09:45:43.003059 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:43.003062 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:43.004143 (+ 1081us) server_negotiation.cc:407] Connection header received
0812 09:45:43.005578 (+ 1435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:43.005581 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:43.005631 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:43.005727 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:43.007047 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:43.007876 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:43.009459 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:43.009794 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:43.009909 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:43.016045 (+ 6136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:43.016049 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:43.016051 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:43.016101 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:43.018753 (+ 2652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:43.019231 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:43.019236 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:43.019239 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:43.019333 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:43.019695 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:43.019698 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:43.019701 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:43.019907 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:43.019995 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:43.022444 (+ 2449us) server_negotiation.cc:299] Negotiation successful
0812 09:45:43.022587 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260812 09:45:43.023496 25024 heartbeater.cc:347] Connected to a master server at 127.13.99.62:43093
I20260812 09:45:43.023558 25021 heartbeater.cc:347] Connected to a master server at 127.13.99.60:38095
I20260812 09:45:43.023779 25021 heartbeater.cc:464] Registering TS with master...
I20260812 09:45:43.023784 25024 heartbeater.cc:464] Registering TS with master...
I20260812 09:45:43.024256 25025 heartbeater.cc:347] Connected to a master server at 127.13.99.61:36161
I20260812 09:45:43.024359 25025 heartbeater.cc:464] Registering TS with master...
I20260812 09:45:43.024454 25021 heartbeater.cc:511] Master 127.13.99.60:38095 requested a full tablet report, sending...
I20260812 09:45:43.024454 25024 heartbeater.cc:511] Master 127.13.99.62:43093 requested a full tablet report, sending...
I20260812 09:45:43.024652 25025 heartbeater.cc:511] Master 127.13.99.61:36161 requested a full tablet report, sending...
I20260812 09:45:43.026130 24737 ts_manager.cc:194] Registered new tserver with Master: 22cbcc6238b04ef196d61a4623d51639 (127.13.99.1:33935)
I20260812 09:45:43.026288 24479 ts_manager.cc:194] Registered new tserver with Master: 22cbcc6238b04ef196d61a4623d51639 (127.13.99.1:33935)
I20260812 09:45:43.027879 23606 ts_manager.cc:194] Re-registered known tserver with Master: 22cbcc6238b04ef196d61a4623d51639 (127.13.99.1:33935)
I20260812 09:45:43.029064 23606 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.99.1@KRBTEST.COM'} at 127.13.99.1:38461
I20260812 09:45:43.041188 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.99.2:46293
--local_ip_for_outbound_sockets=127.13.99.2
--tserver_master_addrs=127.13.99.62:43093
--webserver_port=39009
--webserver_interface=127.13.99.2
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/krb5kdc/oryx_127.13.99.2.keytab
--principal=oryx/127.13.99.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:45813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.13.99.62:43093,127.13.99.61:36161
--tserver_master_addrs=127.13.99.62:43093,127.13.99.61:36161,127.13.99.60:38095 with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 09:45:43.148991 24264 heartbeater.cc:503] Master 127.13.99.62:43093 was elected leader, sending a full tablet report...
W20260812 09:45:43.155697 25033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:45:43.155920 25033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:45:43.155959 25033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:45:43.160090 25033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:45:43.160143 25033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:45:43.160198 25033 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.99.2
I20260812 09:45:43.165165 25033 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:45813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/krb5kdc/oryx_127.13.99.2.keytab
--principal=oryx/127.13.99.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.99.2:46293
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.99.2
--webserver_port=39009
--tserver_master_addrs=127.13.99.62:43093,127.13.99.61:36161,127.13.99.60:38095
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25033
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.99.2
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:45:43.166051 25033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:45:43.166976 25033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:45:43.176645 25045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:45:43.176680 25033 server_base.cc:1034] running on GCE node
W20260812 09:45:43.176680 25040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:43.176923 25041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:45:43.177300 25033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:45:43.177978 25033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:45:43.179196 25033 hybrid_clock.cc:648] HybridClock initialized: now 1786527943179169 us; error 40 us; skew 500 ppm
Aug 12 09:45:43 dist-test-slave-jq4t krb5kdc[23544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527943, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.99.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:45:43.181998 25033 init.cc:377] Logged in from keytab as oryx/127.13.99.2@KRBTEST.COM (short username oryx)
I20260812 09:45:43.182804 25033 webserver.cc:460] Webserver started at http://127.13.99.2:39009/ using document root <none> and password file <none>
I20260812 09:45:43.183348 25033 fs_manager.cc:359] Metadata directory not provided
I20260812 09:45:43.183456 25033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:45:43.187161 25033 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260812 09:45:43.189275 25053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:43.190407 25033 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 09:45:43.190536 25033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/ts-1/wal
uuid: "433ef24880bd41ce9d2092d3748f7792"
format_stamp: "Formatted at 2026-08-12 09:45:41 on dist-test-slave-jq4t"
server_key: "622b55a38e01d16ef2d22fc994b06184"
server_key_iv: "3158c66a005ab6680bb66f082e69f6b0"
server_key_version: "encryptionkey@0"
I20260812 09:45:43.191057 25033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:45:43.209307 25033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:45:43.212575 25033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:45:43.212772 25033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:45:43.213400 25033 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 09:45:43.214279 25033 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:45:43.214354 25033 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:43.214433 25033 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:45:43.214484 25033 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:43.223603 25033 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.2:46293
I20260812 09:45:43.223627 25229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.2:46293 every 8 connection(s)
I20260812 09:45:43.224661 25033 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/ts-1/data/info.pb
I20260812 09:45:43.228624 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 25033
I20260812 09:45:43.228825 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 24087
Aug 12 09:45:43 dist-test-slave-jq4t krb5kdc[23544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527943, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.99.2@KRBTEST.COM for oryx/127.13.99.61@KRBTEST.COM
I20260812 09:45:43.241071 24653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:43.226652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.2:49517 (local address 127.13.99.61:36161)
0812 09:45:43.226839 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:43.226846 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:43.227805 (+ 959us) server_negotiation.cc:407] Connection header received
0812 09:45:43.229112 (+ 1307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:43.229119 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:43.229181 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:43.229233 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:43.230885 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:43.231595 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:43.233177 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:43.233509 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:43.233590 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:43.236726 (+ 3136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:43.236733 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:43.236738 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:43.236768 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:43.238388 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:43.238981 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:43.238987 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:43.238991 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:43.239087 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:43.239488 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:43.239493 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:43.239495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:43.239843 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:43.240006 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:43.240604 (+ 598us) server_negotiation.cc:299] Negotiation successful
0812 09:45:43.240785 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260812 09:45:43.241791 25239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:43.227110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.61:36161 (local address 127.13.99.2:49517)
0812 09:45:43.227650 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:43.227705 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:43.228878 (+ 1173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:43.229601 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:43.229611 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:43.230074 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:45:43.230680 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:43.230698 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:43.231751 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:43.231756 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:43.233039 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:43.233052 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:43.233785 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:43.233790 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:43.233994 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:43.234658 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:45:43.234682 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:45:43.236537 (+ 1855us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:45:43.238528 (+ 1991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:43.238534 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:43.238550 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:43.238875 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:43.239192 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:43.239197 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:43.239201 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:43.239361 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:43.239983 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:45:43.239991 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:45:43.240289 (+ 298us) client_negotiation.cc:769] Sending connection context
0812 09:45:43.240496 (+ 207us) client_negotiation.cc:240] Negotiation successful
0812 09:45:43.240810 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"mutex_wait_us":50,"spinlock_wait_cycles":53120,"thread_start_us":152,"threads_started":1}
I20260812 09:45:43.243495 25235 heartbeater.cc:347] Connected to a master server at 127.13.99.61:36161
I20260812 09:45:43.243726 25235 heartbeater.cc:464] Registering TS with master...
I20260812 09:45:43.244248 25235 heartbeater.cc:511] Master 127.13.99.61:36161 requested a full tablet report, sending...
I20260812 09:45:43.245401 24479 ts_manager.cc:194] Registered new tserver with Master: 433ef24880bd41ce9d2092d3748f7792 (127.13.99.2:46293)
Aug 12 09:45:43 dist-test-slave-jq4t krb5kdc[23544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527943, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.99.2@KRBTEST.COM for oryx/127.13.99.60@KRBTEST.COM
Aug 12 09:45:43 dist-test-slave-jq4t krb5kdc[23544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527943, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.99.2@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM
I20260812 09:45:43.265229 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.99.3:44373
--local_ip_for_outbound_sockets=127.13.99.3
--tserver_master_addrs=127.13.99.62:43093
--webserver_port=36725
--webserver_interface=127.13.99.3
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/krb5kdc/oryx_127.13.99.3.keytab
--principal=oryx/127.13.99.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:45813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.13.99.62:43093,127.13.99.61:36161
--tserver_master_addrs=127.13.99.62:43093,127.13.99.61:36161,127.13.99.60:38095 with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 09:45:43.267567 23658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:43.226796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.2:50829 (local address 127.13.99.62:43093)
0812 09:45:43.226921 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:43.226926 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:43.227806 (+ 880us) server_negotiation.cc:407] Connection header received
0812 09:45:43.257616 (+ 29810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:43.257623 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:43.257696 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:43.257796 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:43.258556 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:43.259195 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:43.261008 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:43.261384 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:43.261528 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:43.263885 (+ 2357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:43.263888 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:43.263891 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:43.263917 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:43.265684 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:43.266205 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:43.266209 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:43.266211 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:43.266301 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:43.266628 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:43.266631 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:43.266633 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:43.266836 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:43.266901 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:43.267274 (+ 373us) server_negotiation.cc:299] Negotiation successful
0812 09:45:43.267388 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260812 09:45:43.267976 25032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:43.228333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.2:35071 (local address 127.13.99.60:38095)
0812 09:45:43.228643 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:43.228647 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:43.228890 (+ 243us) server_negotiation.cc:407] Connection header received
0812 09:45:43.256767 (+ 27877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:43.256773 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:43.256838 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:43.256890 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:43.257951 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:43.258860 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:43.259838 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:43.260097 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:43.260219 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:43.262701 (+ 2482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:43.262705 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:43.262708 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:43.262745 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:43.265861 (+ 3116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:43.266413 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:43.266420 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:43.266424 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:43.266521 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:43.266865 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:43.266871 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:43.266874 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:43.267151 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:43.267235 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:43.267628 (+ 393us) server_negotiation.cc:299] Negotiation successful
0812 09:45:43.267776 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227}
I20260812 09:45:43.268087 25238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:43.227110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:43093 (local address 127.13.99.2:50829)
0812 09:45:43.227651 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:43.227705 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:43.257463 (+ 29758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:43.258147 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:43.258150 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:43.258295 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:45:43.258410 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:43.258415 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:43.259331 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:43.259338 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:43.260813 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:43.260829 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:43.261539 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:43.261546 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:43.261686 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:43.262212 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:45:43.262232 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:45:43.263738 (+ 1506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:45:43.265831 (+ 2093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:43.265835 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:43.265837 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:43.266093 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:43.266416 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:43.266419 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:43.266421 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:43.266509 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:43.266982 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:45:43.266985 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:45:43.267090 (+ 105us) client_negotiation.cc:769] Sending connection context
0812 09:45:43.267748 (+ 658us) client_negotiation.cc:240] Negotiation successful
0812 09:45:43.267890 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"spinlock_wait_cycles":5248,"thread_start_us":140,"threads_started":1}
I20260812 09:45:43.268610 25240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:43.228208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.60:38095 (local address 127.13.99.2:35071)
0812 09:45:43.228804 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:43.228818 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:43.256609 (+ 27791us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:43.257035 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:43.257040 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:43.257304 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:45:43.257453 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:43.257463 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:43.258994 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:43.258997 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:43.259706 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:43.259715 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:43.260226 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:43.260229 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:43.260324 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:43.260943 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:45:43.260981 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:45:43.262556 (+ 1575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:45:43.266012 (+ 3456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:43.266018 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:43.266023 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:43.266289 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:43.266642 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:43.266648 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:43.266654 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:43.266748 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:43.267321 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:45:43.267325 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:45:43.267429 (+ 104us) client_negotiation.cc:769] Sending connection context
0812 09:45:43.268189 (+ 760us) client_negotiation.cc:240] Negotiation successful
0812 09:45:43.268357 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"mutex_wait_us":3,"thread_start_us":94,"threads_started":1}
I20260812 09:45:43.269155 25234 heartbeater.cc:347] Connected to a master server at 127.13.99.62:43093
I20260812 09:45:43.269243 25233 heartbeater.cc:347] Connected to a master server at 127.13.99.60:38095
I20260812 09:45:43.269273 25234 heartbeater.cc:464] Registering TS with master...
I20260812 09:45:43.269358 25233 heartbeater.cc:464] Registering TS with master...
I20260812 09:45:43.269502 25234 heartbeater.cc:511] Master 127.13.99.62:43093 requested a full tablet report, sending...
I20260812 09:45:43.269569 25233 heartbeater.cc:511] Master 127.13.99.60:38095 requested a full tablet report, sending...
I20260812 09:45:43.270213 23606 ts_manager.cc:194] Re-registered known tserver with Master: 433ef24880bd41ce9d2092d3748f7792 (127.13.99.2:46293)
I20260812 09:45:43.270424 24737 ts_manager.cc:194] Registered new tserver with Master: 433ef24880bd41ce9d2092d3748f7792 (127.13.99.2:46293)
I20260812 09:45:43.270560 24417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:43.263829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.60:38095 (local address 127.0.0.1:56356)
0812 09:45:43.264506 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:43.264525 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:43.264609 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:43.265778 (+ 1169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:43.265793 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:43.265819 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:45:43.265928 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:43.265935 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:43.267057 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:43.267060 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:43.268179 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:43.268192 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:43.270211 (+ 2019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:43.270214 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:43.270309 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:43.270332 (+ 23us) client_negotiation.cc:769] Sending connection context
0812 09:45:43.270383 (+ 51us) client_negotiation.cc:240] Negotiation successful
0812 09:45:43.270452 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543}
I20260812 09:45:43.270975 23606 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.99.2@KRBTEST.COM'} at 127.13.99.2:50829
I20260812 09:45:43.271822 25241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:43.264776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56356 (local address 127.13.99.60:38095)
0812 09:45:43.265242 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:43.265247 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:43.265263 (+ 16us) server_negotiation.cc:407] Connection header received
0812 09:45:43.265520 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:43.265524 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:43.265604 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:43.266062 (+ 458us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:45:43.266183 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:43.266921 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:43.269173 (+ 2252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:43.270073 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:43.270986 (+ 913us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:43.271605 (+ 619us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:43.271646 (+ 41us) server_negotiation.cc:299] Negotiation successful
0812 09:45:43.271710 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":86,"threads_started":1}
I20260812 09:45:43.272660 24767 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 443140f48ba94007981c7e2def497963 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260812 09:45:43.273061 24807 sys_catalog.cc:455] T 00000000000000000000000000000000 P 443140f48ba94007981c7e2def497963 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8aaf46e4a6d4531ab509eb45d22e2cd. Latest consensus state: current_term: 1 leader_uuid: "f8aaf46e4a6d4531ab509eb45d22e2cd" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f8aaf46e4a6d4531ab509eb45d22e2cd" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 43093 } } peers { permanent_uuid: "cf7b481d137c42268998cd7a08f2d65e" member_type: VOTER last_known_addr { host: "127.13.99.61" port: 36161 } attrs { promote: false } } peers { permanent_uuid: "443140f48ba94007981c7e2def497963" member_type: NON_VOTER last_known_addr { host: "127.13.99.60" port: 38095 } attrs { promote: true } } }
I20260812 09:45:43.273213 24807 sys_catalog.cc:458] T 00000000000000000000000000000000 P 443140f48ba94007981c7e2def497963 [sys.catalog]: This master's current role is: LEARNER
W20260812 09:45:43.386278 25244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:45:43.386677 25244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:45:43.386760 25244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:45:43.391527 25244 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:45:43.391590 25244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:45:43.391687 25244 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.99.3
I20260812 09:45:43.397470 25244 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:45813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/krb5kdc/oryx_127.13.99.3.keytab
--principal=oryx/127.13.99.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.99.3:44373
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.99.3
--webserver_port=36725
--tserver_master_addrs=127.13.99.62:43093,127.13.99.61:36161,127.13.99.60:38095
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25244
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.99.3
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:45:43.398720 25244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:45:43.399967 25244 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:45:43.411113 25251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:43.411110 25254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:43.411146 25250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:45:43.411415 25244 server_base.cc:1034] running on GCE node
I20260812 09:45:43.411862 25244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:45:43.412621 25244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:45:43.413807 25244 hybrid_clock.cc:648] HybridClock initialized: now 1786527943413794 us; error 33 us; skew 500 ppm
Aug 12 09:45:43 dist-test-slave-jq4t krb5kdc[23544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527943, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.99.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:45:43.416676 25244 init.cc:377] Logged in from keytab as oryx/127.13.99.3@KRBTEST.COM (short username oryx)
I20260812 09:45:43.417639 25244 webserver.cc:460] Webserver started at http://127.13.99.3:36725/ using document root <none> and password file <none>
I20260812 09:45:43.418221 25244 fs_manager.cc:359] Metadata directory not provided
I20260812 09:45:43.418335 25244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:45:43.422281 25244 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 09:45:43.424639 25271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:43.425859 25244 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 09:45:43.425994 25244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/ts-2/wal
uuid: "022e82db4dab49d29d2f320818a7f2f8"
format_stamp: "Formatted at 2026-08-12 09:45:42 on dist-test-slave-jq4t"
server_key: "e033f637072b022d2fe4385147fb0811"
server_key_iv: "82c7e200e0031c7bf6b2782c654a6bc7"
server_key_version: "encryptionkey@0"
I20260812 09:45:43.426507 25244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:45:43.442381 25244 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:45:43.445526 25244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:45:43.445681 25244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:45:43.446197 25244 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 09:45:43.447013 25244 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:45:43.447062 25244 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:43.447100 25244 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:45:43.447115 25244 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:43.456071 25244 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.3:44373
I20260812 09:45:43.456146 25452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.3:44373 every 8 connection(s)
I20260812 09:45:43.457162 25244 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/ts-2/data/info.pb
I20260812 09:45:43.463413 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 25244
Aug 12 09:45:43 dist-test-slave-jq4t krb5kdc[23544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527943, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.99.3@KRBTEST.COM for oryx/127.13.99.61@KRBTEST.COM
Aug 12 09:45:43 dist-test-slave-jq4t krb5kdc[23544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527943, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.99.3@KRBTEST.COM for oryx/127.13.99.60@KRBTEST.COM
Aug 12 09:45:43 dist-test-slave-jq4t krb5kdc[23544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527943, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.99.3@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM
I20260812 09:45:43.475468 25241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:43.460683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.3:46247 (local address 127.13.99.60:38095)
0812 09:45:43.460838 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:43.460843 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:43.460860 (+ 17us) server_negotiation.cc:407] Connection header received
0812 09:45:43.461289 (+ 429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:43.461293 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:43.461347 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:43.461402 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:43.462906 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:43.463716 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:43.465651 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:43.465993 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:43.466073 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:43.470286 (+ 4213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:43.470289 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:43.470293 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:43.470319 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:43.472976 (+ 2657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:43.473610 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:43.473613 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:43.473615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:43.473708 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:43.474127 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:43.474130 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:43.474132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:43.474357 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:43.474438 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:43.475156 (+ 718us) server_negotiation.cc:299] Negotiation successful
0812 09:45:43.475289 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 09:45:43.475999 24653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:43.459027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.3:56569 (local address 127.13.99.61:36161)
0812 09:45:43.459206 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:43.459210 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:43.460064 (+ 854us) server_negotiation.cc:407] Connection header received
0812 09:45:43.461289 (+ 1225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:43.461293 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:43.461347 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:43.461402 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:43.462905 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:43.463716 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:43.465651 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:43.466003 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:43.466073 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:43.470252 (+ 4179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:43.470256 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:43.470259 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:43.470288 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:43.472871 (+ 2583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:43.474618 (+ 1747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:43.474621 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:43.474623 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:43.474710 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:43.475046 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:43.475052 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:43.475057 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:43.475269 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:43.475360 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:43.475702 (+ 342us) server_negotiation.cc:299] Negotiation successful
0812 09:45:43.475843 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260812 09:45:43.476091 23658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:43.458973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.3:53023 (local address 127.13.99.62:43093)
0812 09:45:43.459117 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:43.459122 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:43.460057 (+ 935us) server_negotiation.cc:407] Connection header received
0812 09:45:43.461273 (+ 1216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:43.461277 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:43.461342 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:43.461450 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:43.462883 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:43.463605 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:43.465592 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:43.465903 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:43.465992 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:43.471538 (+ 5546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:43.471543 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:43.471545 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:43.471575 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:43.473395 (+ 1820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:43.474075 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:43.474078 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:43.474080 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:43.474167 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:43.474453 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:43.474456 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:43.474457 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:43.474680 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:43.474790 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:43.475827 (+ 1037us) server_negotiation.cc:299] Negotiation successful
0812 09:45:43.475930 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260812 09:45:43.476595 25464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:43.459409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.61:36161 (local address 127.13.99.3:56569)
0812 09:45:43.459894 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:43.459948 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:43.461076 (+ 1128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:43.461635 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:43.461641 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:43.462114 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:45:43.462727 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:43.462745 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:43.463866 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:43.463870 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:43.465503 (+ 1633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:43.465515 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:43.466254 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:43.466257 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:43.466486 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:43.467314 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:45:43.467337 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:45:43.469537 (+ 2200us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:45:43.473067 (+ 3530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:43.473075 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:43.473092 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:43.473641 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:43.474827 (+ 1186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:43.474832 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:43.474835 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:43.474931 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:43.475415 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:45:43.475418 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:45:43.475500 (+ 82us) client_negotiation.cc:769] Sending connection context
0812 09:45:43.476253 (+ 753us) client_negotiation.cc:240] Negotiation successful
0812 09:45:43.476395 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"mutex_wait_us":509,"spinlock_wait_cycles":48384,"thread_start_us":165,"threads_started":1}
I20260812 09:45:43.476605 25466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:43.460449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.60:38095 (local address 127.13.99.3:46247)
0812 09:45:43.460744 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:43.460760 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:43.461076 (+ 316us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:43.461635 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:43.461641 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:43.462114 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:45:43.462727 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:43.462745 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:43.463866 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:43.463870 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:43.465503 (+ 1633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:43.465515 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:43.466235 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:43.466241 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:43.466552 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:43.467344 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:45:43.467356 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:45:43.470054 (+ 2698us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:45:43.473104 (+ 3050us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:43.473107 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:43.473110 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:43.473479 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:43.473824 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:43.473829 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:43.473831 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:43.474019 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:43.474525 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:45:43.474531 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:45:43.474869 (+ 338us) client_negotiation.cc:769] Sending connection context
0812 09:45:43.475091 (+ 222us) client_negotiation.cc:240] Negotiation successful
0812 09:45:43.475425 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"mutex_wait_us":429,"spinlock_wait_cycles":14976,"thread_start_us":117,"threads_started":1}
I20260812 09:45:43.478012 25456 heartbeater.cc:347] Connected to a master server at 127.13.99.61:36161
I20260812 09:45:43.478278 25456 heartbeater.cc:464] Registering TS with master...
I20260812 09:45:43.478590 25453 heartbeater.cc:347] Connected to a master server at 127.13.99.60:38095
I20260812 09:45:43.478673 25453 heartbeater.cc:464] Registering TS with master...
I20260812 09:45:43.478703 25463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:43.459409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:43093 (local address 127.13.99.3:53023)
0812 09:45:43.459894 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:43.459948 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:43.461060 (+ 1112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:43.461633 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:43.461640 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:43.462110 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:45:43.462726 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:43.462740 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:43.463868 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:43.463872 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:43.465448 (+ 1576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:43.465462 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:43.466218 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:43.466225 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:43.466490 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:43.467426 (+ 936us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:45:43.467444 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:45:43.471403 (+ 3959us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:45:43.473702 (+ 2299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:43.473708 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:43.473712 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:43.473944 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:43.474271 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:43.474274 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:43.474276 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:43.474360 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:43.475157 (+ 797us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:45:43.475161 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:45:43.475251 (+ 90us) client_negotiation.cc:769] Sending connection context
0812 09:45:43.475357 (+ 106us) client_negotiation.cc:240] Negotiation successful
0812 09:45:43.475500 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"mutex_wait_us":257,"spinlock_wait_cycles":66944,"thread_start_us":116,"threads_started":1}
I20260812 09:45:43.478904 25453 heartbeater.cc:511] Master 127.13.99.60:38095 requested a full tablet report, sending...
I20260812 09:45:43.478904 25456 heartbeater.cc:511] Master 127.13.99.61:36161 requested a full tablet report, sending...
I20260812 09:45:43.479527 25455 heartbeater.cc:347] Connected to a master server at 127.13.99.62:43093
I20260812 09:45:43.479614 25455 heartbeater.cc:464] Registering TS with master...
I20260812 09:45:43.479775 25455 heartbeater.cc:511] Master 127.13.99.62:43093 requested a full tablet report, sending...
I20260812 09:45:43.480633 23606 ts_manager.cc:194] Re-registered known tserver with Master: 022e82db4dab49d29d2f320818a7f2f8 (127.13.99.3:44373)
I20260812 09:45:43.480710 24479 ts_manager.cc:194] Registered new tserver with Master: 022e82db4dab49d29d2f320818a7f2f8 (127.13.99.3:44373)
I20260812 09:45:43.480765 24737 ts_manager.cc:194] Registered new tserver with Master: 022e82db4dab49d29d2f320818a7f2f8 (127.13.99.3:44373)
I20260812 09:45:43.481459 23606 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.99.3@KRBTEST.COM'} at 127.13.99.3:53023
I20260812 09:45:43.483359 25032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:43.466302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56358 (local address 127.13.99.60:38095)
0812 09:45:43.466427 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:43.466431 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:43.468393 (+ 1962us) server_negotiation.cc:407] Connection header received
0812 09:45:43.468612 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:43.468617 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:43.468674 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:43.468737 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:43.469668 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:43.470436 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:43.477086 (+ 6650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:43.477510 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:43.477659 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:43.479510 (+ 1851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:43.479514 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:43.479518 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:43.479556 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:43.481344 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:43.481887 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:43.481890 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:43.481892 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:43.481991 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:43.482297 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:43.482300 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:43.482302 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:43.482549 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:43.482688 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:43.483066 (+ 378us) server_negotiation.cc:299] Negotiation successful
0812 09:45:43.483194 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260812 09:45:43.484361 25473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:43.465612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56018 (local address 127.13.99.61:36161)
0812 09:45:43.467934 (+ 2322us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:43.467938 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:43.468194 (+ 256us) server_negotiation.cc:407] Connection header received
0812 09:45:43.470619 (+ 2425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:43.470622 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:43.470673 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:43.470762 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:43.471766 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:43.472544 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:43.473956 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:43.474310 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:43.474420 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:43.477820 (+ 3400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:43.477823 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:43.477826 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:43.477847 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:43.482382 (+ 4535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:43.482855 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:43.482860 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:43.482862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:43.482956 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:43.483283 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:43.483287 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:43.483289 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:43.483532 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:43.483652 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:43.484045 (+ 393us) server_negotiation.cc:299] Negotiation successful
0812 09:45:43.484184 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2250,"thread_start_us":68,"threads_started":1}
I20260812 09:45:43.489048 25476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:43.467600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54550 (local address 127.13.99.62:43093)
0812 09:45:43.469794 (+ 2194us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:43.469798 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:43.469810 (+ 12us) server_negotiation.cc:407] Connection header received
0812 09:45:43.469844 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:43.469847 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:43.469890 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:43.469975 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:43.470931 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:43.482692 (+ 11761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:43.483760 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:43.484121 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:43.485088 (+ 967us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:43.486052 (+ 964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:43.486055 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:43.486057 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:43.486080 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:43.487635 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:43.488016 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:43.488019 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:43.488020 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:43.488093 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:43.488331 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:43.488333 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:43.488334 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:43.488576 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:43.488679 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:43.488830 (+ 151us) server_negotiation.cc:299] Negotiation successful
0812 09:45:43.488934 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2118,"thread_start_us":77,"threads_started":1}
I20260812 09:45:43.491909 23606 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54550:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 09:45:43.495208 23606 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 09:45:43.498108 25481 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8aaf46e4a6d4531ab509eb45d22e2cd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f8aaf46e4a6d4531ab509eb45d22e2cd" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f8aaf46e4a6d4531ab509eb45d22e2cd" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 43093 } } peers { permanent_uuid: "cf7b481d137c42268998cd7a08f2d65e" member_type: VOTER last_known_addr { host: "127.13.99.61" port: 36161 } attrs { promote: false } } peers { permanent_uuid: "443140f48ba94007981c7e2def497963" member_type: NON_VOTER last_known_addr { host: "127.13.99.60" port: 38095 } attrs { promote: true } } }
I20260812 09:45:43.498281 25481 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8aaf46e4a6d4531ab509eb45d22e2cd [sys.catalog]: This master's current role is: LEADER
I20260812 09:45:43.501328 25481 raft_consensus.cc:1050] T 00000000000000000000000000000000 P f8aaf46e4a6d4531ab509eb45d22e2cd: attempting to promote NON_VOTER 443140f48ba94007981c7e2def497963 to VOTER
I20260812 09:45:43.501683 25481 consensus_queue.cc:237] T 00000000000000000000000000000000 P f8aaf46e4a6d4531ab509eb45d22e2cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f8aaf46e4a6d4531ab509eb45d22e2cd" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 43093 } } peers { permanent_uuid: "cf7b481d137c42268998cd7a08f2d65e" member_type: VOTER last_known_addr { host: "127.13.99.61" port: 36161 } attrs { promote: false } } peers { permanent_uuid: "443140f48ba94007981c7e2def497963" member_type: VOTER last_known_addr { host: "127.13.99.60" port: 38095 } attrs { promote: false } }
I20260812 09:45:43.503134 24767 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 443140f48ba94007981c7e2def497963 [term 1 LEARNER]: Refusing update from remote peer f8aaf46e4a6d4531ab509eb45d22e2cd: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260812 09:45:43.503535 25480 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f8aaf46e4a6d4531ab509eb45d22e2cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "443140f48ba94007981c7e2def497963" member_type: VOTER last_known_addr { host: "127.13.99.60" port: 38095 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260812 09:45:43.503598 24492 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cf7b481d137c42268998cd7a08f2d65e [term 1 FOLLOWER]: Refusing update from remote peer f8aaf46e4a6d4531ab509eb45d22e2cd: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260812 09:45:43.504268 25480 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f8aaf46e4a6d4531ab509eb45d22e2cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf7b481d137c42268998cd7a08f2d65e" member_type: VOTER last_known_addr { host: "127.13.99.61" port: 36161 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260812 09:45:43.505865 25481 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f8aaf46e4a6d4531ab509eb45d22e2cd [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 443140f48ba94007981c7e2def497963 (127.13.99.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f8aaf46e4a6d4531ab509eb45d22e2cd" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 43093 } } peers { permanent_uuid: "cf7b481d137c42268998cd7a08f2d65e" member_type: VOTER last_known_addr { host: "127.13.99.61" port: 36161 } attrs { promote: false } } peers { permanent_uuid: "443140f48ba94007981c7e2def497963" member_type: VOTER last_known_addr { host: "127.13.99.60" port: 38095 } attrs { promote: false } } }
I20260812 09:45:43.506421 24492 raft_consensus.cc:2932] T 00000000000000000000000000000000 P cf7b481d137c42268998cd7a08f2d65e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 443140f48ba94007981c7e2def497963 (127.13.99.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f8aaf46e4a6d4531ab509eb45d22e2cd" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 43093 } } peers { permanent_uuid: "cf7b481d137c42268998cd7a08f2d65e" member_type: VOTER last_known_addr { host: "127.13.99.61" port: 36161 } attrs { promote: false } } peers { permanent_uuid: "443140f48ba94007981c7e2def497963" member_type: VOTER last_known_addr { host: "127.13.99.60" port: 38095 } attrs { promote: false } } }
I20260812 09:45:43.506457 24767 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 443140f48ba94007981c7e2def497963 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 443140f48ba94007981c7e2def497963 (127.13.99.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f8aaf46e4a6d4531ab509eb45d22e2cd" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 43093 } } peers { permanent_uuid: "cf7b481d137c42268998cd7a08f2d65e" member_type: VOTER last_known_addr { host: "127.13.99.61" port: 36161 } attrs { promote: false } } peers { permanent_uuid: "443140f48ba94007981c7e2def497963" member_type: VOTER last_known_addr { host: "127.13.99.60" port: 38095 } attrs { promote: false } } }
I20260812 09:45:43.507405 25481 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8aaf46e4a6d4531ab509eb45d22e2cd [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f8aaf46e4a6d4531ab509eb45d22e2cd" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f8aaf46e4a6d4531ab509eb45d22e2cd" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 43093 } } peers { permanent_uuid: "cf7b481d137c42268998cd7a08f2d65e" member_type: VOTER last_known_addr { host: "127.13.99.61" port: 36161 } attrs { promote: false } } peers { permanent_uuid: "443140f48ba94007981c7e2def497963" member_type: VOTER last_known_addr { host: "127.13.99.60" port: 38095 } attrs { promote: false } } }
I20260812 09:45:43.507511 25481 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8aaf46e4a6d4531ab509eb45d22e2cd [sys.catalog]: This master's current role is: LEADER
I20260812 09:45:43.507836 25480 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8aaf46e4a6d4531ab509eb45d22e2cd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f8aaf46e4a6d4531ab509eb45d22e2cd" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f8aaf46e4a6d4531ab509eb45d22e2cd" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 43093 } } peers { permanent_uuid: "cf7b481d137c42268998cd7a08f2d65e" member_type: VOTER last_known_addr { host: "127.13.99.61" port: 36161 } attrs { promote: false } } peers { permanent_uuid: "443140f48ba94007981c7e2def497963" member_type: VOTER last_known_addr { host: "127.13.99.60" port: 38095 } attrs { promote: false } } }
I20260812 09:45:43.507956 25480 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8aaf46e4a6d4531ab509eb45d22e2cd [sys.catalog]: This master's current role is: LEADER
I20260812 09:45:43.508034 24694 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8aaf46e4a6d4531ab509eb45d22e2cd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f8aaf46e4a6d4531ab509eb45d22e2cd" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f8aaf46e4a6d4531ab509eb45d22e2cd" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 43093 } } peers { permanent_uuid: "cf7b481d137c42268998cd7a08f2d65e" member_type: VOTER last_known_addr { host: "127.13.99.61" port: 36161 } attrs { promote: false } } peers { permanent_uuid: "443140f48ba94007981c7e2def497963" member_type: VOTER last_known_addr { host: "127.13.99.60" port: 38095 } attrs { promote: false } } }
I20260812 09:45:43.508116 24694 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8aaf46e4a6d4531ab509eb45d22e2cd [sys.catalog]: This master's current role is: LEADER
I20260812 09:45:43.508534 24807 sys_catalog.cc:455] T 00000000000000000000000000000000 P 443140f48ba94007981c7e2def497963 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f8aaf46e4a6d4531ab509eb45d22e2cd" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f8aaf46e4a6d4531ab509eb45d22e2cd" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 43093 } } peers { permanent_uuid: "cf7b481d137c42268998cd7a08f2d65e" member_type: VOTER last_known_addr { host: "127.13.99.61" port: 36161 } attrs { promote: false } } peers { permanent_uuid: "443140f48ba94007981c7e2def497963" member_type: VOTER last_known_addr { host: "127.13.99.60" port: 38095 } attrs { promote: false } } }
I20260812 09:45:43.508653 24807 sys_catalog.cc:458] T 00000000000000000000000000000000 P 443140f48ba94007981c7e2def497963 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 09:45:43.512061 25496 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf7b481d137c42268998cd7a08f2d65e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f8aaf46e4a6d4531ab509eb45d22e2cd" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f8aaf46e4a6d4531ab509eb45d22e2cd" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 43093 } } peers { permanent_uuid: "cf7b481d137c42268998cd7a08f2d65e" member_type: VOTER last_known_addr { host: "127.13.99.61" port: 36161 } attrs { promote: false } } peers { permanent_uuid: "443140f48ba94007981c7e2def497963" member_type: VOTER last_known_addr { host: "127.13.99.60" port: 38095 } attrs { promote: false } } }
I20260812 09:45:43.512269 25496 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf7b481d137c42268998cd7a08f2d65e [sys.catalog]: This master's current role is: FOLLOWER
I20260812 09:45:43.517234 24417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:43.510739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.1:33935 (local address 127.0.0.1:45724)
0812 09:45:43.510886 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:43.510903 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:43.510989 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:43.513186 (+ 2197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:43.513189 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:43.513207 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:45:43.513323 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:43.513332 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:43.514839 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:43.514842 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:43.515712 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:43.515723 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:43.516825 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:43.516829 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:43.516916 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:43.516942 (+ 26us) client_negotiation.cc:769] Sending connection context
0812 09:45:43.517006 (+ 64us) client_negotiation.cc:240] Negotiation successful
0812 09:45:43.517071 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260812 09:45:43.517295 25498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:43.510961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45724 (local address 127.13.99.1:33935)
0812 09:45:43.512689 (+ 1728us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:43.512708 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:43.512733 (+ 25us) server_negotiation.cc:407] Connection header received
0812 09:45:43.512826 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:43.512831 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:43.513050 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:43.513224 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:45:43.513466 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:43.514701 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:43.515842 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:43.516688 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:43.516814 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:43.516967 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:43.517037 (+ 70us) server_negotiation.cc:299] Negotiation successful
0812 09:45:43.517139 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":140,"threads_started":1}
I20260812 09:45:43.517717 25500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:43.511395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.2:46293 (local address 127.0.0.1:36968)
0812 09:45:43.513466 (+ 2071us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:43.513479 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:43.513572 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:43.514031 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:43.514035 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:43.514054 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:45:43.514137 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:43.514144 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:43.515205 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:43.515208 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:43.516058 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:43.516069 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:43.517306 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:43.517312 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:43.517413 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:43.517445 (+ 32us) client_negotiation.cc:769] Sending connection context
0812 09:45:43.517520 (+ 75us) client_negotiation.cc:240] Negotiation successful
0812 09:45:43.517593 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2004,"thread_start_us":106,"threads_started":1}
I20260812 09:45:43.518231 25506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:43.511977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36968 (local address 127.13.99.2:46293)
0812 09:45:43.512304 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:43.512317 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:43.513671 (+ 1354us) server_negotiation.cc:407] Connection header received
0812 09:45:43.513757 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:43.513761 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:43.513892 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:43.514048 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:45:43.514233 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:43.515086 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:43.516198 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:43.516968 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:43.517111 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:43.517235 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:43.517984 (+ 749us) server_negotiation.cc:299] Negotiation successful
0812 09:45:43.518088 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":139,"threads_started":1}
I20260812 09:45:43.519037 25499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:43.511036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.3:44373 (local address 127.0.0.1:57446)
0812 09:45:43.511805 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:43.511818 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:43.511924 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:43.513459 (+ 1535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:43.513462 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:43.513481 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:45:43.513584 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:43.513591 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:43.516000 (+ 2409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:43.516003 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:43.517082 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:43.517091 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:43.518628 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:43.518631 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:43.518722 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:43.518746 (+ 24us) client_negotiation.cc:769] Sending connection context
0812 09:45:43.518838 (+ 92us) client_negotiation.cc:240] Negotiation successful
0812 09:45:43.518906 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":123,"threads_started":1}
I20260812 09:45:43.519148 25502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:43.511494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57446 (local address 127.13.99.3:44373)
0812 09:45:43.511821 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:43.511834 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:43.511894 (+ 60us) server_negotiation.cc:407] Connection header received
0812 09:45:43.513066 (+ 1172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:43.513096 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:43.513301 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:43.513457 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:45:43.513674 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:43.514760 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:43.517455 (+ 2695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:43.518274 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:43.518385 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:43.518560 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:43.518887 (+ 327us) server_negotiation.cc:299] Negotiation successful
0812 09:45:43.518990 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":98,"threads_started":1}
I20260812 09:45:43.520960 24918 tablet_service.cc:1467] Processing CreateTablet for tablet 7c740152d0a3465093f05612bfafd3af (DEFAULT_TABLE table=test-table [id=876f7c48cb5c4023bb7a51a1fce24175]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 09:45:43.522045 24918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c740152d0a3465093f05612bfafd3af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:45:43.521783 25347 tablet_service.cc:1467] Processing CreateTablet for tablet 7c740152d0a3465093f05612bfafd3af (DEFAULT_TABLE table=test-table [id=876f7c48cb5c4023bb7a51a1fce24175]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 09:45:43.522477 25134 tablet_service.cc:1467] Processing CreateTablet for tablet 7c740152d0a3465093f05612bfafd3af (DEFAULT_TABLE table=test-table [id=876f7c48cb5c4023bb7a51a1fce24175]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 09:45:43.522924 25347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c740152d0a3465093f05612bfafd3af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:45:43.523485 25134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c740152d0a3465093f05612bfafd3af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:45:43.529500 25508 tablet_bootstrap.cc:492] T 7c740152d0a3465093f05612bfafd3af P 22cbcc6238b04ef196d61a4623d51639: Bootstrap starting.
I20260812 09:45:43.532200 25509 tablet_bootstrap.cc:492] T 7c740152d0a3465093f05612bfafd3af P 022e82db4dab49d29d2f320818a7f2f8: Bootstrap starting.
I20260812 09:45:43.532980 25508 tablet_bootstrap.cc:654] T 7c740152d0a3465093f05612bfafd3af P 22cbcc6238b04ef196d61a4623d51639: Neither blocks nor log segments found. Creating new log.
I20260812 09:45:43.533790 25508 log.cc:824] T 7c740152d0a3465093f05612bfafd3af P 22cbcc6238b04ef196d61a4623d51639: Log is configured to *not* fsync() on all Append() calls
I20260812 09:45:43.534233 25509 tablet_bootstrap.cc:654] T 7c740152d0a3465093f05612bfafd3af P 022e82db4dab49d29d2f320818a7f2f8: Neither blocks nor log segments found. Creating new log.
I20260812 09:45:43.534969 25509 log.cc:824] T 7c740152d0a3465093f05612bfafd3af P 022e82db4dab49d29d2f320818a7f2f8: Log is configured to *not* fsync() on all Append() calls
I20260812 09:45:43.535687 25511 tablet_bootstrap.cc:492] T 7c740152d0a3465093f05612bfafd3af P 433ef24880bd41ce9d2092d3748f7792: Bootstrap starting.
I20260812 09:45:43.535779 25508 tablet_bootstrap.cc:492] T 7c740152d0a3465093f05612bfafd3af P 22cbcc6238b04ef196d61a4623d51639: No bootstrap required, opened a new log
I20260812 09:45:43.535956 25508 ts_tablet_manager.cc:1397] T 7c740152d0a3465093f05612bfafd3af P 22cbcc6238b04ef196d61a4623d51639: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260812 09:45:43.537721 25511 tablet_bootstrap.cc:654] T 7c740152d0a3465093f05612bfafd3af P 433ef24880bd41ce9d2092d3748f7792: Neither blocks nor log segments found. Creating new log.
I20260812 09:45:43.537855 25509 tablet_bootstrap.cc:492] T 7c740152d0a3465093f05612bfafd3af P 022e82db4dab49d29d2f320818a7f2f8: No bootstrap required, opened a new log
I20260812 09:45:43.538095 25509 ts_tablet_manager.cc:1397] T 7c740152d0a3465093f05612bfafd3af P 022e82db4dab49d29d2f320818a7f2f8: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260812 09:45:43.538522 25511 log.cc:824] T 7c740152d0a3465093f05612bfafd3af P 433ef24880bd41ce9d2092d3748f7792: Log is configured to *not* fsync() on all Append() calls
I20260812 09:45:43.539915 25508 raft_consensus.cc:348] T 7c740152d0a3465093f05612bfafd3af P 22cbcc6238b04ef196d61a4623d51639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22cbcc6238b04ef196d61a4623d51639" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 33935 } } peers { permanent_uuid: "022e82db4dab49d29d2f320818a7f2f8" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 44373 } } peers { permanent_uuid: "433ef24880bd41ce9d2092d3748f7792" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 46293 } }
I20260812 09:45:43.540169 25508 raft_consensus.cc:374] T 7c740152d0a3465093f05612bfafd3af P 22cbcc6238b04ef196d61a4623d51639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:45:43.540211 25508 raft_consensus.cc:729] T 7c740152d0a3465093f05612bfafd3af P 22cbcc6238b04ef196d61a4623d51639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22cbcc6238b04ef196d61a4623d51639, State: Initialized, Role: FOLLOWER
I20260812 09:45:43.540652 25508 consensus_queue.cc:260] T 7c740152d0a3465093f05612bfafd3af P 22cbcc6238b04ef196d61a4623d51639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22cbcc6238b04ef196d61a4623d51639" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 33935 } } peers { permanent_uuid: "022e82db4dab49d29d2f320818a7f2f8" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 44373 } } peers { permanent_uuid: "433ef24880bd41ce9d2092d3748f7792" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 46293 } }
I20260812 09:45:43.541548 25508 ts_tablet_manager.cc:1428] T 7c740152d0a3465093f05612bfafd3af P 22cbcc6238b04ef196d61a4623d51639: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260812 09:45:43.542361 25511 tablet_bootstrap.cc:492] T 7c740152d0a3465093f05612bfafd3af P 433ef24880bd41ce9d2092d3748f7792: No bootstrap required, opened a new log
I20260812 09:45:43.542292 25509 raft_consensus.cc:348] T 7c740152d0a3465093f05612bfafd3af P 022e82db4dab49d29d2f320818a7f2f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22cbcc6238b04ef196d61a4623d51639" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 33935 } } peers { permanent_uuid: "022e82db4dab49d29d2f320818a7f2f8" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 44373 } } peers { permanent_uuid: "433ef24880bd41ce9d2092d3748f7792" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 46293 } }
I20260812 09:45:43.542519 25511 ts_tablet_manager.cc:1397] T 7c740152d0a3465093f05612bfafd3af P 433ef24880bd41ce9d2092d3748f7792: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260812 09:45:43.542537 25509 raft_consensus.cc:374] T 7c740152d0a3465093f05612bfafd3af P 022e82db4dab49d29d2f320818a7f2f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:45:43.542586 25509 raft_consensus.cc:729] T 7c740152d0a3465093f05612bfafd3af P 022e82db4dab49d29d2f320818a7f2f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 022e82db4dab49d29d2f320818a7f2f8, State: Initialized, Role: FOLLOWER
I20260812 09:45:43.543000 25509 consensus_queue.cc:260] T 7c740152d0a3465093f05612bfafd3af P 022e82db4dab49d29d2f320818a7f2f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22cbcc6238b04ef196d61a4623d51639" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 33935 } } peers { permanent_uuid: "022e82db4dab49d29d2f320818a7f2f8" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 44373 } } peers { permanent_uuid: "433ef24880bd41ce9d2092d3748f7792" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 46293 } }
I20260812 09:45:43.544102 25509 ts_tablet_manager.cc:1428] T 7c740152d0a3465093f05612bfafd3af P 022e82db4dab49d29d2f320818a7f2f8: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260812 09:45:43.544523 25455 heartbeater.cc:503] Master 127.13.99.62:43093 was elected leader, sending a full tablet report...
I20260812 09:45:43.545265 25024 heartbeater.cc:503] Master 127.13.99.62:43093 was elected leader, sending a full tablet report...
I20260812 09:45:43.547256 25511 raft_consensus.cc:348] T 7c740152d0a3465093f05612bfafd3af P 433ef24880bd41ce9d2092d3748f7792 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22cbcc6238b04ef196d61a4623d51639" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 33935 } } peers { permanent_uuid: "022e82db4dab49d29d2f320818a7f2f8" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 44373 } } peers { permanent_uuid: "433ef24880bd41ce9d2092d3748f7792" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 46293 } }
I20260812 09:45:43.547572 25511 raft_consensus.cc:374] T 7c740152d0a3465093f05612bfafd3af P 433ef24880bd41ce9d2092d3748f7792 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:45:43.547644 25511 raft_consensus.cc:729] T 7c740152d0a3465093f05612bfafd3af P 433ef24880bd41ce9d2092d3748f7792 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 433ef24880bd41ce9d2092d3748f7792, State: Initialized, Role: FOLLOWER
I20260812 09:45:43.548061 25511 consensus_queue.cc:260] T 7c740152d0a3465093f05612bfafd3af P 433ef24880bd41ce9d2092d3748f7792 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22cbcc6238b04ef196d61a4623d51639" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 33935 } } peers { permanent_uuid: "022e82db4dab49d29d2f320818a7f2f8" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 44373 } } peers { permanent_uuid: "433ef24880bd41ce9d2092d3748f7792" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 46293 } }
I20260812 09:45:43.548730 25234 heartbeater.cc:503] Master 127.13.99.62:43093 was elected leader, sending a full tablet report...
I20260812 09:45:43.549052 25511 ts_tablet_manager.cc:1428] T 7c740152d0a3465093f05612bfafd3af P 433ef24880bd41ce9d2092d3748f7792: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
W20260812 09:45:43.709048 25460 tablet.cc:2369] T 7c740152d0a3465093f05612bfafd3af P 022e82db4dab49d29d2f320818a7f2f8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 09:45:43.726527 25236 tablet.cc:2369] T 7c740152d0a3465093f05612bfafd3af P 433ef24880bd41ce9d2092d3748f7792: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 09:45:43.753505 25027 tablet.cc:2369] T 7c740152d0a3465093f05612bfafd3af P 22cbcc6238b04ef196d61a4623d51639: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 09:45:43.838729 25516 raft_consensus.cc:482] T 7c740152d0a3465093f05612bfafd3af P 22cbcc6238b04ef196d61a4623d51639 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 09:45:43.838982 25516 raft_consensus.cc:504] T 7c740152d0a3465093f05612bfafd3af P 22cbcc6238b04ef196d61a4623d51639 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22cbcc6238b04ef196d61a4623d51639" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 33935 } } peers { permanent_uuid: "022e82db4dab49d29d2f320818a7f2f8" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 44373 } } peers { permanent_uuid: "433ef24880bd41ce9d2092d3748f7792" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 46293 } }
I20260812 09:45:43.840309 25516 leader_election.cc:288] T 7c740152d0a3465093f05612bfafd3af P 22cbcc6238b04ef196d61a4623d51639 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 022e82db4dab49d29d2f320818a7f2f8 (127.13.99.3:44373), 433ef24880bd41ce9d2092d3748f7792 (127.13.99.2:46293)
I20260812 09:45:43.844748 25506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:43.840839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.1:50795 (local address 127.13.99.2:46293)
0812 09:45:43.840990 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:43.840996 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:43.841132 (+ 136us) server_negotiation.cc:407] Connection header received
0812 09:45:43.841295 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:43.841297 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:43.841342 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:43.841545 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:45:43.841875 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:43.842378 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:43.843540 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:43.844116 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:43.844199 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:43.844279 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:43.844527 (+ 248us) server_negotiation.cc:299] Negotiation successful
0812 09:45:43.844619 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260812 09:45:43.844802 25502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:43.840655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.1:60021 (local address 127.13.99.3:44373)
0812 09:45:43.840790 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:43.840798 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:43.841054 (+ 256us) server_negotiation.cc:407] Connection header received
0812 09:45:43.841260 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:43.841264 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:43.841324 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:43.841385 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:45:43.841838 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:43.842385 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:43.843620 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:43.844180 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:43.844292 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:43.844354 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:43.844615 (+ 261us) server_negotiation.cc:299] Negotiation successful
0812 09:45:43.844692 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260812 09:45:43.844772 25524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:43.840760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.2:46293 (local address 127.13.99.1:50795)
0812 09:45:43.841039 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:43.841052 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:43.841152 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:43.841574 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:43.841576 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:43.841601 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:45:43.841737 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:43.841742 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:43.842494 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:43.842499 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:43.843394 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:43.843403 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:43.844241 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:43.844247 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:43.844346 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:43.844378 (+ 32us) client_negotiation.cc:769] Sending connection context
0812 09:45:43.844475 (+ 97us) client_negotiation.cc:240] Negotiation successful
0812 09:45:43.844593 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"mutex_wait_us":41,"spinlock_wait_cycles":768,"thread_start_us":103,"threads_started":1}
I20260812 09:45:43.844790 25522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:43.840533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.3:44373 (local address 127.13.99.1:60021)
0812 09:45:43.840956 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:43.840978 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:43.841112 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:43.841529 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:43.841534 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:43.841551 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:45:43.841721 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:43.841727 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:43.842530 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:43.842534 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:43.843399 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:43.843410 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:43.844314 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:43.844318 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:43.844410 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:43.844429 (+ 19us) client_negotiation.cc:769] Sending connection context
0812 09:45:43.844563 (+ 134us) client_negotiation.cc:240] Negotiation successful
0812 09:45:43.844630 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"mutex_wait_us":55,"spinlock_wait_cycles":5248,"thread_start_us":117,"threads_started":1}
I20260812 09:45:43.845706 25164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c740152d0a3465093f05612bfafd3af" candidate_uuid: "22cbcc6238b04ef196d61a4623d51639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "433ef24880bd41ce9d2092d3748f7792" is_pre_election: true
I20260812 09:45:43.845715 25378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c740152d0a3465093f05612bfafd3af" candidate_uuid: "22cbcc6238b04ef196d61a4623d51639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "022e82db4dab49d29d2f320818a7f2f8" is_pre_election: true
I20260812 09:45:43.846019 25378 raft_consensus.cc:2454] T 7c740152d0a3465093f05612bfafd3af P 022e82db4dab49d29d2f320818a7f2f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22cbcc6238b04ef196d61a4623d51639 in term 0.
I20260812 09:45:43.846027 25164 raft_consensus.cc:2454] T 7c740152d0a3465093f05612bfafd3af P 433ef24880bd41ce9d2092d3748f7792 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22cbcc6238b04ef196d61a4623d51639 in term 0.
I20260812 09:45:43.846529 24851 leader_election.cc:302] T 7c740152d0a3465093f05612bfafd3af P 22cbcc6238b04ef196d61a4623d51639 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 022e82db4dab49d29d2f320818a7f2f8, 22cbcc6238b04ef196d61a4623d51639; no voters:
I20260812 09:45:43.846837 25516 raft_consensus.cc:2781] T 7c740152d0a3465093f05612bfafd3af P 22cbcc6238b04ef196d61a4623d51639 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 09:45:43.846925 25516 raft_consensus.cc:482] T 7c740152d0a3465093f05612bfafd3af P 22cbcc6238b04ef196d61a4623d51639 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 09:45:43.846956 25516 raft_consensus.cc:3037] T 7c740152d0a3465093f05612bfafd3af P 22cbcc6238b04ef196d61a4623d51639 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:45:43.848095 25516 raft_consensus.cc:504] T 7c740152d0a3465093f05612bfafd3af P 22cbcc6238b04ef196d61a4623d51639 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22cbcc6238b04ef196d61a4623d51639" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 33935 } } peers { permanent_uuid: "022e82db4dab49d29d2f320818a7f2f8" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 44373 } } peers { permanent_uuid: "433ef24880bd41ce9d2092d3748f7792" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 46293 } }
I20260812 09:45:43.848513 25516 leader_election.cc:288] T 7c740152d0a3465093f05612bfafd3af P 22cbcc6238b04ef196d61a4623d51639 [CANDIDATE]: Term 1 election: Requested vote from peers 022e82db4dab49d29d2f320818a7f2f8 (127.13.99.3:44373), 433ef24880bd41ce9d2092d3748f7792 (127.13.99.2:46293)
I20260812 09:45:43.848867 25164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c740152d0a3465093f05612bfafd3af" candidate_uuid: "22cbcc6238b04ef196d61a4623d51639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "433ef24880bd41ce9d2092d3748f7792"
I20260812 09:45:43.848888 25378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c740152d0a3465093f05612bfafd3af" candidate_uuid: "22cbcc6238b04ef196d61a4623d51639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "022e82db4dab49d29d2f320818a7f2f8"
I20260812 09:45:43.849004 25164 raft_consensus.cc:3037] T 7c740152d0a3465093f05612bfafd3af P 433ef24880bd41ce9d2092d3748f7792 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:45:43.849016 25378 raft_consensus.cc:3037] T 7c740152d0a3465093f05612bfafd3af P 022e82db4dab49d29d2f320818a7f2f8 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:45:43.850008 25164 raft_consensus.cc:2454] T 7c740152d0a3465093f05612bfafd3af P 433ef24880bd41ce9d2092d3748f7792 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22cbcc6238b04ef196d61a4623d51639 in term 1.
I20260812 09:45:43.850086 25378 raft_consensus.cc:2454] T 7c740152d0a3465093f05612bfafd3af P 022e82db4dab49d29d2f320818a7f2f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22cbcc6238b04ef196d61a4623d51639 in term 1.
I20260812 09:45:43.850394 24849 leader_election.cc:302] T 7c740152d0a3465093f05612bfafd3af P 22cbcc6238b04ef196d61a4623d51639 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22cbcc6238b04ef196d61a4623d51639, 433ef24880bd41ce9d2092d3748f7792; no voters:
I20260812 09:45:43.850613 25516 raft_consensus.cc:2781] T 7c740152d0a3465093f05612bfafd3af P 22cbcc6238b04ef196d61a4623d51639 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 09:45:43.850855 25516 raft_consensus.cc:686] T 7c740152d0a3465093f05612bfafd3af P 22cbcc6238b04ef196d61a4623d51639 [term 1 LEADER]: Becoming Leader. State: Replica: 22cbcc6238b04ef196d61a4623d51639, State: Running, Role: LEADER
I20260812 09:45:43.851392 25516 consensus_queue.cc:237] T 7c740152d0a3465093f05612bfafd3af P 22cbcc6238b04ef196d61a4623d51639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22cbcc6238b04ef196d61a4623d51639" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 33935 } } peers { permanent_uuid: "022e82db4dab49d29d2f320818a7f2f8" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 44373 } } peers { permanent_uuid: "433ef24880bd41ce9d2092d3748f7792" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 46293 } }
I20260812 09:45:43.854962 23606 catalog_manager.cc:5495] T 7c740152d0a3465093f05612bfafd3af P 22cbcc6238b04ef196d61a4623d51639 reported cstate change: term changed from 0 to 1, leader changed from <none> to 22cbcc6238b04ef196d61a4623d51639 (127.13.99.1). New cstate: current_term: 1 leader_uuid: "22cbcc6238b04ef196d61a4623d51639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22cbcc6238b04ef196d61a4623d51639" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 33935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "022e82db4dab49d29d2f320818a7f2f8" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 44373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "433ef24880bd41ce9d2092d3748f7792" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 46293 } health_report { overall_health: UNKNOWN } } }
I20260812 09:45:43.895243 25498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:43.891337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45728 (local address 127.13.99.1:33935)
0812 09:45:43.891509 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:43.891516 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:43.891532 (+ 16us) server_negotiation.cc:407] Connection header received
0812 09:45:43.891628 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:43.891630 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:43.891687 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:43.891809 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 09:45:43.892094 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:43.892698 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:43.893686 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:43.893927 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:43.894045 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:43.894399 (+ 354us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 09:45:43.894832 (+ 433us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 09:45:43.894962 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:43.895033 (+ 71us) server_negotiation.cc:299] Negotiation successful
0812 09:45:43.895093 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260812 09:45:43.901296 25378 raft_consensus.cc:1261] T 7c740152d0a3465093f05612bfafd3af P 022e82db4dab49d29d2f320818a7f2f8 [term 1 FOLLOWER]: Refusing update from remote peer 22cbcc6238b04ef196d61a4623d51639: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 09:45:43.901294 25164 raft_consensus.cc:1261] T 7c740152d0a3465093f05612bfafd3af P 433ef24880bd41ce9d2092d3748f7792 [term 1 FOLLOWER]: Refusing update from remote peer 22cbcc6238b04ef196d61a4623d51639: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 09:45:43.902114 25525 consensus_queue.cc:1035] T 7c740152d0a3465093f05612bfafd3af P 22cbcc6238b04ef196d61a4623d51639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "433ef24880bd41ce9d2092d3748f7792" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 46293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 09:45:43.902310 25516 consensus_queue.cc:1035] T 7c740152d0a3465093f05612bfafd3af P 22cbcc6238b04ef196d61a4623d51639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "022e82db4dab49d29d2f320818a7f2f8" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 44373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 09:45:43.910036 25530 mvcc.cc:205] Tried to move back new op lower bound from 7317618458212134912 to 7317618458017927168. Current Snapshot: MvccSnapshot[applied={T|T < 7317618458212134912}]
I20260812 09:45:43.916390 25531 mvcc.cc:205] Tried to move back new op lower bound from 7317618458212134912 to 7317618458017927168. Current Snapshot: MvccSnapshot[applied={T|T < 7317618458212134912}]
I20260812 09:45:43.925565 23604 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54550:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 09:45:43.925776 23604 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54550:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 09:45:43.931077 23604 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f8aaf46e4a6d4531ab509eb45d22e2cd: Sending DeleteTablet for 3 replicas of tablet 7c740152d0a3465093f05612bfafd3af
I20260812 09:45:43.931813 24918 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c740152d0a3465093f05612bfafd3af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 09:45:43 UTC) from {username='oryx', principal='oryx/127.13.99.62@KRBTEST.COM'} at 127.0.0.1:45724
I20260812 09:45:43.931860 25347 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c740152d0a3465093f05612bfafd3af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 09:45:43 UTC) from {username='oryx', principal='oryx/127.13.99.62@KRBTEST.COM'} at 127.0.0.1:57446
I20260812 09:45:43.932061 25134 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c740152d0a3465093f05612bfafd3af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 09:45:43 UTC) from {username='oryx', principal='oryx/127.13.99.62@KRBTEST.COM'} at 127.0.0.1:36968
I20260812 09:45:43.932554 25545 tablet_replica.cc:330] T 7c740152d0a3465093f05612bfafd3af P 433ef24880bd41ce9d2092d3748f7792: stopping tablet replica
I20260812 09:45:43.932893 25545 raft_consensus.cc:2229] T 7c740152d0a3465093f05612bfafd3af P 433ef24880bd41ce9d2092d3748f7792 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 09:45:43.933231 25545 raft_consensus.cc:2258] T 7c740152d0a3465093f05612bfafd3af P 433ef24880bd41ce9d2092d3748f7792 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 09:45:43.935115 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 23562
I20260812 09:45:43.935120 25545 ts_tablet_manager.cc:1905] T 7c740152d0a3465093f05612bfafd3af P 433ef24880bd41ce9d2092d3748f7792: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 09:45:43.942159 25545 ts_tablet_manager.cc:1918] T 7c740152d0a3465093f05612bfafd3af P 433ef24880bd41ce9d2092d3748f7792: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 09:45:43.942401 25545 log.cc:1196] T 7c740152d0a3465093f05612bfafd3af P 433ef24880bd41ce9d2092d3748f7792: Deleting WAL directory at /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/ts-1/wal/wals/7c740152d0a3465093f05612bfafd3af
I20260812 09:45:43.943011 25545 ts_tablet_manager.cc:1939] T 7c740152d0a3465093f05612bfafd3af P 433ef24880bd41ce9d2092d3748f7792: Deleting consensus metadata
I20260812 09:45:43.950168 25473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:43.933809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56030 (local address 127.13.99.61:36161)
0812 09:45:43.933972 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:43.933978 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:43.933993 (+ 15us) server_negotiation.cc:407] Connection header received
0812 09:45:43.934037 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:43.934039 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:43.934111 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:43.934218 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:43.935222 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:43.935863 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:43.938964 (+ 3101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:43.939322 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:43.939400 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:43.945690 (+ 6290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:43.945698 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:43.945704 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:43.945741 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:43.947835 (+ 2094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:43.948402 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:43.948409 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:43.948414 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:43.948581 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:43.948922 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:43.948928 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:43.948933 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:43.949254 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:43.949430 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:43.949712 (+ 282us) server_negotiation.cc:299] Negotiation successful
0812 09:45:43.949903 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
W20260812 09:45:43.959165 25055 connection.cc:535] server connection from 127.0.0.1:36968 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 09:45:43.959519 25032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:43.933851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56374 (local address 127.13.99.60:38095)
0812 09:45:43.933959 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:43.933964 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:43.950149 (+ 16185us) server_negotiation.cc:407] Connection header received
0812 09:45:43.950349 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:43.950353 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:43.950412 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:43.950466 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:43.951395 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:43.952226 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:43.953442 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:43.953895 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:43.954021 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:43.955642 (+ 1621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:43.955649 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:43.955655 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:43.955691 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:43.957544 (+ 1853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:43.958049 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:43.958053 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:43.958055 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:43.958148 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:43.958470 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:43.958477 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:43.958481 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:43.958784 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:43.958936 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:43.959210 (+ 274us) server_negotiation.cc:299] Negotiation successful
0812 09:45:43.959355 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260812 09:45:43.961112 25542 tablet_replica.cc:330] T 7c740152d0a3465093f05612bfafd3af P 022e82db4dab49d29d2f320818a7f2f8: stopping tablet replica
I20260812 09:45:43.961459 25542 raft_consensus.cc:2229] T 7c740152d0a3465093f05612bfafd3af P 022e82db4dab49d29d2f320818a7f2f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 09:45:43.961819 25542 raft_consensus.cc:2258] T 7c740152d0a3465093f05612bfafd3af P 022e82db4dab49d29d2f320818a7f2f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 09:45:43.962261 25544 tablet_replica.cc:330] T 7c740152d0a3465093f05612bfafd3af P 22cbcc6238b04ef196d61a4623d51639: stopping tablet replica
I20260812 09:45:43.962574 25544 raft_consensus.cc:2229] T 7c740152d0a3465093f05612bfafd3af P 22cbcc6238b04ef196d61a4623d51639 [term 1 LEADER]: Raft consensus shutting down.
I20260812 09:45:43.963071 25544 raft_consensus.cc:2258] T 7c740152d0a3465093f05612bfafd3af P 22cbcc6238b04ef196d61a4623d51639 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 09:45:43.964901 25544 ts_tablet_manager.cc:1905] T 7c740152d0a3465093f05612bfafd3af P 22cbcc6238b04ef196d61a4623d51639: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 09:45:43.965518 25542 ts_tablet_manager.cc:1905] T 7c740152d0a3465093f05612bfafd3af P 022e82db4dab49d29d2f320818a7f2f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 09:45:43.967963 25544 ts_tablet_manager.cc:1918] T 7c740152d0a3465093f05612bfafd3af P 22cbcc6238b04ef196d61a4623d51639: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 09:45:43.968070 25544 log.cc:1196] T 7c740152d0a3465093f05612bfafd3af P 22cbcc6238b04ef196d61a4623d51639: Deleting WAL directory at /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/ts-0/wal/wals/7c740152d0a3465093f05612bfafd3af
I20260812 09:45:43.968371 25542 ts_tablet_manager.cc:1918] T 7c740152d0a3465093f05612bfafd3af P 022e82db4dab49d29d2f320818a7f2f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 09:45:43.968415 25544 ts_tablet_manager.cc:1939] T 7c740152d0a3465093f05612bfafd3af P 22cbcc6238b04ef196d61a4623d51639: Deleting consensus metadata
I20260812 09:45:43.968462 25542 log.cc:1196] T 7c740152d0a3465093f05612bfafd3af P 022e82db4dab49d29d2f320818a7f2f8: Deleting WAL directory at /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/ts-2/wal/wals/7c740152d0a3465093f05612bfafd3af
I20260812 09:45:43.969066 25542 ts_tablet_manager.cc:1939] T 7c740152d0a3465093f05612bfafd3af P 022e82db4dab49d29d2f320818a7f2f8: Deleting consensus metadata
W20260812 09:45:43.969579 24851 connection.cc:441] server connection from 127.0.0.1:45724 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:45724 (request call id 1) could send its response
W20260812 09:45:43.970304 25273 connection.cc:441] server connection from 127.0.0.1:57446 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57446 (request call id 1) could send its response
I20260812 09:45:43.978089 25473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:43.964447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56032 (local address 127.13.99.61:36161)
0812 09:45:43.964608 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:43.964612 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:43.968091 (+ 3479us) server_negotiation.cc:407] Connection header received
0812 09:45:43.968280 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:43.968284 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:43.968335 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:43.968399 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:43.969430 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:43.970240 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:43.971495 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:43.971842 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:43.971915 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:43.973853 (+ 1938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:43.973859 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:43.973865 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:43.973899 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:43.975571 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:43.976067 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:43.976071 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:43.976073 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:43.976183 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:43.976713 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:43.976717 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:43.976719 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:43.977075 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:43.977227 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:43.977733 (+ 506us) server_negotiation.cc:299] Negotiation successful
0812 09:45:43.977890 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260812 09:45:43.979034 25032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:43.965160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56388 (local address 127.13.99.60:38095)
0812 09:45:43.965289 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:43.965294 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:43.970027 (+ 4733us) server_negotiation.cc:407] Connection header received
0812 09:45:43.970215 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:43.970218 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:43.970271 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:43.970329 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:43.971367 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:43.971976 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:43.973174 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:43.973474 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:43.973602 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:43.975260 (+ 1658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:43.975264 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:43.975267 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:43.975298 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:43.976969 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:43.977534 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:43.977537 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:43.977539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:43.977663 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:43.978058 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:43.978062 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:43.978064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:43.978299 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:43.978428 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:43.978759 (+ 331us) server_negotiation.cc:299] Negotiation successful
0812 09:45:43.978883 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260812 09:45:44.916128 25557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:44.915636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:43093 (local address 127.13.99.2:49181)
0812 09:45:44.915902 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:44.916013 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:43093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":72,"threads_started":1}
W20260812 09:45:44.916700 25234 heartbeater.cc:644] Failed to heartbeat to 127.13.99.62:43093 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.13.99.62:43093: connect: Connection refused (error 111)
I20260812 09:45:44.918583 25560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:44.918211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:43093 (local address 127.13.99.3:44177)
0812 09:45:44.918428 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:44.918504 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:43093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":85,"threads_started":1}
W20260812 09:45:44.918856 25455 heartbeater.cc:644] Failed to heartbeat to 127.13.99.62:43093 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.13.99.62:43093: connect: Connection refused (error 111)
I20260812 09:45:44.920341 25561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:44.918267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:43093 (local address 127.13.99.1:41559)
0812 09:45:44.920176 (+ 1909us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:44.920262 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:43093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1842,"thread_start_us":66,"threads_started":1}
W20260812 09:45:44.920835 25024 heartbeater.cc:644] Failed to heartbeat to 127.13.99.62:43093 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.13.99.62:43093: connect: Connection refused (error 111)
I20260812 09:45:45.430598 25563 raft_consensus.cc:482] T 00000000000000000000000000000000 P 443140f48ba94007981c7e2def497963 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8aaf46e4a6d4531ab509eb45d22e2cd)
I20260812 09:45:45.430981 25563 raft_consensus.cc:504] T 00000000000000000000000000000000 P 443140f48ba94007981c7e2def497963 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f8aaf46e4a6d4531ab509eb45d22e2cd" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 43093 } } peers { permanent_uuid: "cf7b481d137c42268998cd7a08f2d65e" member_type: VOTER last_known_addr { host: "127.13.99.61" port: 36161 } attrs { promote: false } } peers { permanent_uuid: "443140f48ba94007981c7e2def497963" member_type: VOTER last_known_addr { host: "127.13.99.60" port: 38095 } attrs { promote: false } }
I20260812 09:45:45.431334 25564 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf7b481d137c42268998cd7a08f2d65e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8aaf46e4a6d4531ab509eb45d22e2cd)
I20260812 09:45:45.431520 25564 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf7b481d137c42268998cd7a08f2d65e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f8aaf46e4a6d4531ab509eb45d22e2cd" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 43093 } } peers { permanent_uuid: "cf7b481d137c42268998cd7a08f2d65e" member_type: VOTER last_known_addr { host: "127.13.99.61" port: 36161 } attrs { promote: false } } peers { permanent_uuid: "443140f48ba94007981c7e2def497963" member_type: VOTER last_known_addr { host: "127.13.99.60" port: 38095 } attrs { promote: false } }
I20260812 09:45:45.432199 25563 leader_election.cc:288] T 00000000000000000000000000000000 P 443140f48ba94007981c7e2def497963 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8aaf46e4a6d4531ab509eb45d22e2cd (127.13.99.62:43093), cf7b481d137c42268998cd7a08f2d65e (127.13.99.61:36161)
I20260812 09:45:45.432452 25564 leader_election.cc:288] T 00000000000000000000000000000000 P cf7b481d137c42268998cd7a08f2d65e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8aaf46e4a6d4531ab509eb45d22e2cd (127.13.99.62:43093), 443140f48ba94007981c7e2def497963 (127.13.99.60:38095)
I20260812 09:45:45.432951 25565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:45.432336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:43093 (local address 127.0.0.1:48676)
0812 09:45:45.432738 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:45.432840 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:43093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":75,"threads_started":1}
I20260812 09:45:45.433265 25569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:45.432585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:43093 (local address 127.0.0.1:48690)
0812 09:45:45.432979 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:45.433167 (+ 188us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:43093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":72,"threads_started":1}
W20260812 09:45:45.433321 24728 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.99.62:43093: connect: Connection refused (error 111)
W20260812 09:45:45.433517 24457 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.99.62:43093: connect: Connection refused (error 111)
I20260812 09:45:45.434396 25565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:45.434073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:43093 (local address 127.0.0.1:48702)
0812 09:45:45.434263 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:45.434314 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:43093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
W20260812 09:45:45.434506 24728 leader_election.cc:334] T 00000000000000000000000000000000 P 443140f48ba94007981c7e2def497963 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f8aaf46e4a6d4531ab509eb45d22e2cd (127.13.99.62:43093): Network error: Client connection negotiation failed: client connection to 127.13.99.62:43093: connect: Connection refused (error 111)
I20260812 09:45:45.434736 25569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:45.434301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:43093 (local address 127.0.0.1:48708)
0812 09:45:45.434519 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:45.434666 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:43093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260812 09:45:45.434886 24457 leader_election.cc:334] T 00000000000000000000000000000000 P cf7b481d137c42268998cd7a08f2d65e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f8aaf46e4a6d4531ab509eb45d22e2cd (127.13.99.62:43093): Network error: Client connection negotiation failed: client connection to 127.13.99.62:43093: connect: Connection refused (error 111)
I20260812 09:45:45.437232 25571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:45.432759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.61:36161 (local address 127.0.0.1:42210)
0812 09:45:45.433024 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:45.433039 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:45.433152 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:45.433672 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:45.433675 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:45.433690 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:45:45.433870 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:45.433878 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:45.435010 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:45.435017 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:45.435964 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:45.435976 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:45.436867 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:45.436871 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:45.436961 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:45.436987 (+ 26us) client_negotiation.cc:769] Sending connection context
0812 09:45:45.437043 (+ 56us) client_negotiation.cc:240] Negotiation successful
0812 09:45:45.437112 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":71,"threads_started":1}
I20260812 09:45:45.437372 25573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:45.433082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46806 (local address 127.13.99.60:38095)
0812 09:45:45.433306 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:45.433311 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:45.433583 (+ 272us) server_negotiation.cc:407] Connection header received
0812 09:45:45.433635 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:45.433640 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:45.433688 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:45.433758 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:45:45.434091 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:45.434878 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:45.436115 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:45.436703 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:45.436815 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:45.436890 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:45.437200 (+ 310us) server_negotiation.cc:299] Negotiation successful
0812 09:45:45.437272 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":76,"threads_started":1}
I20260812 09:45:45.437372 25570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:45.432748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.60:38095 (local address 127.0.0.1:46806)
0812 09:45:45.433387 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:45.433402 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:45.433499 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:45.433840 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:45.433843 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:45.433862 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:45:45.433975 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:45.433981 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:45.435063 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:45.435069 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:45.435961 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:45.435975 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:45.436951 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:45.436955 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:45.437058 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:45.437079 (+ 21us) client_negotiation.cc:769] Sending connection context
0812 09:45:45.437170 (+ 91us) client_negotiation.cc:240] Negotiation successful
0812 09:45:45.437241 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":58,"threads_started":1}
I20260812 09:45:45.437767 25568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:45.432543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42210 (local address 127.13.99.61:36161)
0812 09:45:45.433444 (+ 901us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:45.433450 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:45.433464 (+ 14us) server_negotiation.cc:407] Connection header received
0812 09:45:45.433507 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:45.433510 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:45.433555 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:45.433978 (+ 423us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:45:45.434066 (+ 88us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:45.434882 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:45.436123 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:45.436724 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:45.437443 (+ 719us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:45.437513 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:45.437552 (+ 39us) server_negotiation.cc:299] Negotiation successful
0812 09:45:45.437621 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":830,"thread_start_us":72,"threads_started":1}
I20260812 09:45:45.437813 24767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cf7b481d137c42268998cd7a08f2d65e" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "443140f48ba94007981c7e2def497963" is_pre_election: true
I20260812 09:45:45.438045 24767 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 443140f48ba94007981c7e2def497963 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf7b481d137c42268998cd7a08f2d65e in term 1.
I20260812 09:45:45.438205 24492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "443140f48ba94007981c7e2def497963" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "cf7b481d137c42268998cd7a08f2d65e" is_pre_election: true
I20260812 09:45:45.438411 24492 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cf7b481d137c42268998cd7a08f2d65e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 443140f48ba94007981c7e2def497963 in term 1.
I20260812 09:45:45.438429 24460 leader_election.cc:302] T 00000000000000000000000000000000 P cf7b481d137c42268998cd7a08f2d65e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 443140f48ba94007981c7e2def497963, cf7b481d137c42268998cd7a08f2d65e; no voters: f8aaf46e4a6d4531ab509eb45d22e2cd
I20260812 09:45:45.438647 25564 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf7b481d137c42268998cd7a08f2d65e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260812 09:45:45.438695 25564 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf7b481d137c42268998cd7a08f2d65e [term 1 FOLLOWER]: Starting leader election (detected failure of leader f8aaf46e4a6d4531ab509eb45d22e2cd)
I20260812 09:45:45.438723 25564 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf7b481d137c42268998cd7a08f2d65e [term 1 FOLLOWER]: Advancing to term 2
I20260812 09:45:45.438968 24730 leader_election.cc:302] T 00000000000000000000000000000000 P 443140f48ba94007981c7e2def497963 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 443140f48ba94007981c7e2def497963, cf7b481d137c42268998cd7a08f2d65e; no voters: f8aaf46e4a6d4531ab509eb45d22e2cd
I20260812 09:45:45.439172 25563 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 443140f48ba94007981c7e2def497963 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260812 09:45:45.439229 25563 raft_consensus.cc:482] T 00000000000000000000000000000000 P 443140f48ba94007981c7e2def497963 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f8aaf46e4a6d4531ab509eb45d22e2cd)
I20260812 09:45:45.439284 25563 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 443140f48ba94007981c7e2def497963 [term 1 FOLLOWER]: Advancing to term 2
I20260812 09:45:45.439924 25564 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf7b481d137c42268998cd7a08f2d65e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f8aaf46e4a6d4531ab509eb45d22e2cd" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 43093 } } peers { permanent_uuid: "cf7b481d137c42268998cd7a08f2d65e" member_type: VOTER last_known_addr { host: "127.13.99.61" port: 36161 } attrs { promote: false } } peers { permanent_uuid: "443140f48ba94007981c7e2def497963" member_type: VOTER last_known_addr { host: "127.13.99.60" port: 38095 } attrs { promote: false } }
I20260812 09:45:45.440294 25564 leader_election.cc:288] T 00000000000000000000000000000000 P cf7b481d137c42268998cd7a08f2d65e [CANDIDATE]: Term 2 election: Requested vote from peers f8aaf46e4a6d4531ab509eb45d22e2cd (127.13.99.62:43093), 443140f48ba94007981c7e2def497963 (127.13.99.60:38095)
I20260812 09:45:45.440269 25563 raft_consensus.cc:504] T 00000000000000000000000000000000 P 443140f48ba94007981c7e2def497963 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f8aaf46e4a6d4531ab509eb45d22e2cd" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 43093 } } peers { permanent_uuid: "cf7b481d137c42268998cd7a08f2d65e" member_type: VOTER last_known_addr { host: "127.13.99.61" port: 36161 } attrs { promote: false } } peers { permanent_uuid: "443140f48ba94007981c7e2def497963" member_type: VOTER last_known_addr { host: "127.13.99.60" port: 38095 } attrs { promote: false } }
I20260812 09:45:45.440618 24767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cf7b481d137c42268998cd7a08f2d65e" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "443140f48ba94007981c7e2def497963"
I20260812 09:45:45.440671 25563 leader_election.cc:288] T 00000000000000000000000000000000 P 443140f48ba94007981c7e2def497963 [CANDIDATE]: Term 2 election: Requested vote from peers f8aaf46e4a6d4531ab509eb45d22e2cd (127.13.99.62:43093), cf7b481d137c42268998cd7a08f2d65e (127.13.99.61:36161)
I20260812 09:45:45.440775 24767 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 443140f48ba94007981c7e2def497963 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate cf7b481d137c42268998cd7a08f2d65e in current term 2: Already voted for candidate 443140f48ba94007981c7e2def497963 in this term.
I20260812 09:45:45.440865 25570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:45.440414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:43093 (local address 127.0.0.1:48724)
0812 09:45:45.440609 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:45.440684 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:43093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260812 09:45:45.440963 25571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:45.440752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:43093 (local address 127.0.0.1:48740)
0812 09:45:45.440843 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:45.440904 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:43093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260812 09:45:45.441234 24492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "443140f48ba94007981c7e2def497963" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "cf7b481d137c42268998cd7a08f2d65e"
I20260812 09:45:45.441404 24492 raft_consensus.cc:2379] T 00000000000000000000000000000000 P cf7b481d137c42268998cd7a08f2d65e [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 443140f48ba94007981c7e2def497963 in current term 2: Already voted for candidate cf7b481d137c42268998cd7a08f2d65e in this term.
I20260812 09:45:45.441687 25570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:45.441466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:43093 (local address 127.0.0.1:48752)
0812 09:45:45.441569 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:45.441628 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:43093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260812 09:45:45.441910 24457 leader_election.cc:334] T 00000000000000000000000000000000 P cf7b481d137c42268998cd7a08f2d65e [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f8aaf46e4a6d4531ab509eb45d22e2cd (127.13.99.62:43093): Network error: Client connection negotiation failed: client connection to 127.13.99.62:43093: connect: Connection refused (error 111)
I20260812 09:45:45.441969 25571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:45.441689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:43093 (local address 127.0.0.1:48762)
0812 09:45:45.441824 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:45.441912 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:43093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260812 09:45:45.441990 24457 leader_election.cc:302] T 00000000000000000000000000000000 P cf7b481d137c42268998cd7a08f2d65e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cf7b481d137c42268998cd7a08f2d65e; no voters: 443140f48ba94007981c7e2def497963, f8aaf46e4a6d4531ab509eb45d22e2cd
I20260812 09:45:45.442137 25564 raft_consensus.cc:2726] T 00000000000000000000000000000000 P cf7b481d137c42268998cd7a08f2d65e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260812 09:45:45.442270 24728 leader_election.cc:334] T 00000000000000000000000000000000 P 443140f48ba94007981c7e2def497963 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f8aaf46e4a6d4531ab509eb45d22e2cd (127.13.99.62:43093): Network error: Client connection negotiation failed: client connection to 127.13.99.62:43093: connect: Connection refused (error 111)
I20260812 09:45:45.442363 24728 leader_election.cc:302] T 00000000000000000000000000000000 P 443140f48ba94007981c7e2def497963 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443140f48ba94007981c7e2def497963; no voters: cf7b481d137c42268998cd7a08f2d65e, f8aaf46e4a6d4531ab509eb45d22e2cd
I20260812 09:45:45.442489 25563 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 443140f48ba94007981c7e2def497963 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260812 09:45:45.917702 25579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:45.917234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:43093 (local address 127.13.99.2:38391)
0812 09:45:45.917512 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:45.917571 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:43093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":96,"threads_started":1}
I20260812 09:45:45.919831 25581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:45.919454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:43093 (local address 127.13.99.3:43481)
0812 09:45:45.919694 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:45.919748 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:43093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":66,"threads_started":1}
I20260812 09:45:45.921680 25582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:45.921316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:43093 (local address 127.13.99.1:60921)
0812 09:45:45.921558 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:45.921607 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:43093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":101,"threads_started":1}
I20260812 09:45:46.918897 25583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:46.918421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:43093 (local address 127.13.99.2:53211)
0812 09:45:46.918681 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:46.918779 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:43093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":99,"threads_started":1}
I20260812 09:45:46.920888 25585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:46.920547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:43093 (local address 127.13.99.3:43805)
0812 09:45:46.920758 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:46.920812 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:43093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":57,"threads_started":1}
I20260812 09:45:46.922626 25586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:46.922279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:43093 (local address 127.13.99.1:50905)
0812 09:45:46.922499 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:46.922550 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:43093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":65,"threads_started":1}
I20260812 09:45:47.296341 25587 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf7b481d137c42268998cd7a08f2d65e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 09:45:47.296553 25587 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf7b481d137c42268998cd7a08f2d65e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f8aaf46e4a6d4531ab509eb45d22e2cd" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 43093 } } peers { permanent_uuid: "cf7b481d137c42268998cd7a08f2d65e" member_type: VOTER last_known_addr { host: "127.13.99.61" port: 36161 } attrs { promote: false } } peers { permanent_uuid: "443140f48ba94007981c7e2def497963" member_type: VOTER last_known_addr { host: "127.13.99.60" port: 38095 } attrs { promote: false } }
I20260812 09:45:47.296968 25587 leader_election.cc:288] T 00000000000000000000000000000000 P cf7b481d137c42268998cd7a08f2d65e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f8aaf46e4a6d4531ab509eb45d22e2cd (127.13.99.62:43093), 443140f48ba94007981c7e2def497963 (127.13.99.60:38095)
I20260812 09:45:47.297441 24767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cf7b481d137c42268998cd7a08f2d65e" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "443140f48ba94007981c7e2def497963" is_pre_election: true
I20260812 09:45:47.297576 24767 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 443140f48ba94007981c7e2def497963 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf7b481d137c42268998cd7a08f2d65e in term 2.
I20260812 09:45:47.297629 25589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:47.297143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:43093 (local address 127.0.0.1:48958)
0812 09:45:47.297442 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:47.297498 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:43093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":71,"threads_started":1}
I20260812 09:45:47.298074 24460 leader_election.cc:302] T 00000000000000000000000000000000 P cf7b481d137c42268998cd7a08f2d65e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 443140f48ba94007981c7e2def497963, cf7b481d137c42268998cd7a08f2d65e; no voters:
I20260812 09:45:47.298269 25587 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf7b481d137c42268998cd7a08f2d65e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260812 09:45:47.298346 25587 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf7b481d137c42268998cd7a08f2d65e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 09:45:47.298435 25587 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf7b481d137c42268998cd7a08f2d65e [term 2 FOLLOWER]: Advancing to term 3
I20260812 09:45:47.298590 25589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:47.298344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:43093 (local address 127.0.0.1:48960)
0812 09:45:47.298463 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:47.298522 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:43093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260812 09:45:47.298717 24457 leader_election.cc:334] T 00000000000000000000000000000000 P cf7b481d137c42268998cd7a08f2d65e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f8aaf46e4a6d4531ab509eb45d22e2cd (127.13.99.62:43093): Network error: Client connection negotiation failed: client connection to 127.13.99.62:43093: connect: Connection refused (error 111)
I20260812 09:45:47.299594 25587 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf7b481d137c42268998cd7a08f2d65e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f8aaf46e4a6d4531ab509eb45d22e2cd" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 43093 } } peers { permanent_uuid: "cf7b481d137c42268998cd7a08f2d65e" member_type: VOTER last_known_addr { host: "127.13.99.61" port: 36161 } attrs { promote: false } } peers { permanent_uuid: "443140f48ba94007981c7e2def497963" member_type: VOTER last_known_addr { host: "127.13.99.60" port: 38095 } attrs { promote: false } }
I20260812 09:45:47.299948 25587 leader_election.cc:288] T 00000000000000000000000000000000 P cf7b481d137c42268998cd7a08f2d65e [CANDIDATE]: Term 3 election: Requested vote from peers f8aaf46e4a6d4531ab509eb45d22e2cd (127.13.99.62:43093), 443140f48ba94007981c7e2def497963 (127.13.99.60:38095)
I20260812 09:45:47.300287 25589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:47.300061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:43093 (local address 127.0.0.1:48966)
0812 09:45:47.300163 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:47.300213 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:43093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260812 09:45:47.300300 24767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cf7b481d137c42268998cd7a08f2d65e" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "443140f48ba94007981c7e2def497963"
I20260812 09:45:47.300439 24767 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 443140f48ba94007981c7e2def497963 [term 2 FOLLOWER]: Advancing to term 3
I20260812 09:45:47.301054 25589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:47.300856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:43093 (local address 127.0.0.1:48976)
0812 09:45:47.300965 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:47.301008 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:43093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260812 09:45:47.301215 24457 leader_election.cc:334] T 00000000000000000000000000000000 P cf7b481d137c42268998cd7a08f2d65e [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer f8aaf46e4a6d4531ab509eb45d22e2cd (127.13.99.62:43093): Network error: Client connection negotiation failed: client connection to 127.13.99.62:43093: connect: Connection refused (error 111)
I20260812 09:45:47.301662 24767 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 443140f48ba94007981c7e2def497963 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf7b481d137c42268998cd7a08f2d65e in term 3.
I20260812 09:45:47.301942 24460 leader_election.cc:302] T 00000000000000000000000000000000 P cf7b481d137c42268998cd7a08f2d65e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 443140f48ba94007981c7e2def497963, cf7b481d137c42268998cd7a08f2d65e; no voters: f8aaf46e4a6d4531ab509eb45d22e2cd
I20260812 09:45:47.302142 25587 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf7b481d137c42268998cd7a08f2d65e [term 3 FOLLOWER]: Leader election won for term 3
I20260812 09:45:47.302330 25587 raft_consensus.cc:686] T 00000000000000000000000000000000 P cf7b481d137c42268998cd7a08f2d65e [term 3 LEADER]: Becoming Leader. State: Replica: cf7b481d137c42268998cd7a08f2d65e, State: Running, Role: LEADER
I20260812 09:45:47.302598 25587 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf7b481d137c42268998cd7a08f2d65e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f8aaf46e4a6d4531ab509eb45d22e2cd" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 43093 } } peers { permanent_uuid: "cf7b481d137c42268998cd7a08f2d65e" member_type: VOTER last_known_addr { host: "127.13.99.61" port: 36161 } attrs { promote: false } } peers { permanent_uuid: "443140f48ba94007981c7e2def497963" member_type: VOTER last_known_addr { host: "127.13.99.60" port: 38095 } attrs { promote: false } }
I20260812 09:45:47.303934 25592 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf7b481d137c42268998cd7a08f2d65e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf7b481d137c42268998cd7a08f2d65e. Latest consensus state: current_term: 3 leader_uuid: "cf7b481d137c42268998cd7a08f2d65e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f8aaf46e4a6d4531ab509eb45d22e2cd" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 43093 } } peers { permanent_uuid: "cf7b481d137c42268998cd7a08f2d65e" member_type: VOTER last_known_addr { host: "127.13.99.61" port: 36161 } attrs { promote: false } } peers { permanent_uuid: "443140f48ba94007981c7e2def497963" member_type: VOTER last_known_addr { host: "127.13.99.60" port: 38095 } attrs { promote: false } } }
I20260812 09:45:47.304031 25592 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf7b481d137c42268998cd7a08f2d65e [sys.catalog]: This master's current role is: LEADER
I20260812 09:45:47.304324 25595 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 09:45:47.306767 25595 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 09:45:47.307173 25595 catalog_manager.cc:1259] Loaded cluster ID: 5feee61a67a142d8bd855e4f84785f79
I20260812 09:45:47.307246 25595 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 09:45:47.307808 25595 catalog_manager.cc:1504] Loading token signing keys...
I20260812 09:45:47.308202 25595 catalog_manager.cc:5879] T 00000000000000000000000000000000 P cf7b481d137c42268998cd7a08f2d65e: Loaded TSK: 0
I20260812 09:45:47.309016 25595 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 09:45:47.356685 24478 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56032:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 09:45:47.359813 24478 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 09:45:47.362927 25589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:47.362644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:43093 (local address 127.0.0.1:48980)
0812 09:45:47.362776 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:47.362838 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:43093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260812 09:45:47.363034 24767 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 443140f48ba94007981c7e2def497963 [term 3 FOLLOWER]: Refusing update from remote peer cf7b481d137c42268998cd7a08f2d65e: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260812 09:45:47.363554 25592 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cf7b481d137c42268998cd7a08f2d65e [LEADER]: Connected to new peer: Peer: permanent_uuid: "443140f48ba94007981c7e2def497963" member_type: VOTER last_known_addr { host: "127.13.99.60" port: 38095 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260812 09:45:47.363914 25589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:47.363651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:43093 (local address 127.0.0.1:48982)
0812 09:45:47.363789 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:47.363846 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:43093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260812 09:45:47.364022 24457 consensus_peers.cc:487] T 00000000000000000000000000000000 P cf7b481d137c42268998cd7a08f2d65e -> Peer f8aaf46e4a6d4531ab509eb45d22e2cd (127.13.99.62:43093): Couldn't send request to peer f8aaf46e4a6d4531ab509eb45d22e2cd. Status: Network error: Client connection negotiation failed: client connection to 127.13.99.62:43093: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260812 09:45:47.365149 25597 sys_catalog.cc:455] T 00000000000000000000000000000000 P 443140f48ba94007981c7e2def497963 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf7b481d137c42268998cd7a08f2d65e. Latest consensus state: current_term: 3 leader_uuid: "cf7b481d137c42268998cd7a08f2d65e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f8aaf46e4a6d4531ab509eb45d22e2cd" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 43093 } } peers { permanent_uuid: "cf7b481d137c42268998cd7a08f2d65e" member_type: VOTER last_known_addr { host: "127.13.99.61" port: 36161 } attrs { promote: false } } peers { permanent_uuid: "443140f48ba94007981c7e2def497963" member_type: VOTER last_known_addr { host: "127.13.99.60" port: 38095 } attrs { promote: false } } }
I20260812 09:45:47.365264 25597 sys_catalog.cc:458] T 00000000000000000000000000000000 P 443140f48ba94007981c7e2def497963 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 09:45:47.366394 25587 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf7b481d137c42268998cd7a08f2d65e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "cf7b481d137c42268998cd7a08f2d65e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f8aaf46e4a6d4531ab509eb45d22e2cd" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 43093 } } peers { permanent_uuid: "cf7b481d137c42268998cd7a08f2d65e" member_type: VOTER last_known_addr { host: "127.13.99.61" port: 36161 } attrs { promote: false } } peers { permanent_uuid: "443140f48ba94007981c7e2def497963" member_type: VOTER last_known_addr { host: "127.13.99.60" port: 38095 } attrs { promote: false } } }
I20260812 09:45:47.366516 25587 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf7b481d137c42268998cd7a08f2d65e [sys.catalog]: This master's current role is: LEADER
I20260812 09:45:47.366760 25597 sys_catalog.cc:455] T 00000000000000000000000000000000 P 443140f48ba94007981c7e2def497963 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "cf7b481d137c42268998cd7a08f2d65e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f8aaf46e4a6d4531ab509eb45d22e2cd" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 43093 } } peers { permanent_uuid: "cf7b481d137c42268998cd7a08f2d65e" member_type: VOTER last_known_addr { host: "127.13.99.61" port: 36161 } attrs { promote: false } } peers { permanent_uuid: "443140f48ba94007981c7e2def497963" member_type: VOTER last_known_addr { host: "127.13.99.60" port: 38095 } attrs { promote: false } } }
I20260812 09:45:47.366916 25597 sys_catalog.cc:458] T 00000000000000000000000000000000 P 443140f48ba94007981c7e2def497963 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 09:45:47.377337 25589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:47.372417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.1:33935 (local address 127.0.0.1:56536)
0812 09:45:47.372541 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:47.372555 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:47.372641 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:47.373204 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:47.373207 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:47.373227 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:45:47.373339 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:47.373345 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:47.374466 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:47.374470 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:47.375414 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:47.375423 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:47.376927 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:47.376932 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:47.377029 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:47.377069 (+ 40us) client_negotiation.cc:769] Sending connection context
0812 09:45:47.377140 (+ 71us) client_negotiation.cc:240] Negotiation successful
0812 09:45:47.377220 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260812 09:45:47.377556 25604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:47.372948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.2:46293 (local address 127.0.0.1:59100)
0812 09:45:47.373187 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:47.373203 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:47.373305 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:47.373969 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:47.373972 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:47.373988 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:45:47.374076 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:47.374081 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:47.375140 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:47.375144 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:47.376245 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:47.376255 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:47.377136 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:47.377139 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:47.377258 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:47.377282 (+ 24us) client_negotiation.cc:769] Sending connection context
0812 09:45:47.377347 (+ 65us) client_negotiation.cc:240] Negotiation successful
0812 09:45:47.377419 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"mutex_wait_us":18,"thread_start_us":109,"threads_started":1}
I20260812 09:45:47.377570 25606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:47.373325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59100 (local address 127.13.99.2:46293)
0812 09:45:47.373626 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:47.373632 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:47.373647 (+ 15us) server_negotiation.cc:407] Connection header received
0812 09:45:47.373772 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:47.373775 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:47.373830 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:47.373962 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:45:47.374196 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:47.375026 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:47.376383 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:47.377014 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:47.377270 (+ 256us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:47.377336 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:47.377373 (+ 37us) server_negotiation.cc:299] Negotiation successful
0812 09:45:47.377438 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":112,"threads_started":1}
I20260812 09:45:47.377938 25603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:47.372561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56536 (local address 127.13.99.1:33935)
0812 09:45:47.372919 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:47.372924 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:47.372939 (+ 15us) server_negotiation.cc:407] Connection header received
0812 09:45:47.372998 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:47.373004 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:47.373067 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:47.373458 (+ 391us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:45:47.373594 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:47.374376 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:47.375574 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:47.376796 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:47.376904 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:47.376977 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:47.377619 (+ 642us) server_negotiation.cc:299] Negotiation successful
0812 09:45:47.377731 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":95,"threads_started":1}
I20260812 09:45:47.378190 25134 tablet_service.cc:1467] Processing CreateTablet for tablet 3b9f4604cc984ca284a7f2586229fef2 (DEFAULT_TABLE table=test-table [id=93402c0d19304ecf89f21b61786d2b4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 09:45:47.378363 25605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:47.373000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.3:44373 (local address 127.0.0.1:47166)
0812 09:45:47.373676 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:47.373690 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:47.373777 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:47.374342 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:47.374345 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:47.374372 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:45:47.374476 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:47.374482 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:47.375424 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:47.375429 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:47.376201 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:47.376210 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:47.377094 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:47.377101 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:47.378062 (+ 961us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:47.378087 (+ 25us) client_negotiation.cc:769] Sending connection context
0812 09:45:47.378163 (+ 76us) client_negotiation.cc:240] Negotiation successful
0812 09:45:47.378240 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"mutex_wait_us":25,"thread_start_us":103,"threads_started":1}
I20260812 09:45:47.378556 25134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b9f4604cc984ca284a7f2586229fef2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:45:47.378571 25607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:47.373389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47166 (local address 127.13.99.3:44373)
0812 09:45:47.374109 (+ 720us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:47.374114 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:47.374128 (+ 14us) server_negotiation.cc:407] Connection header received
0812 09:45:47.374179 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:47.374182 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:47.374225 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:47.374313 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:45:47.374619 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:47.375308 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:47.376336 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:47.376978 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:47.377088 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:47.377170 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:47.378189 (+ 1019us) server_negotiation.cc:299] Negotiation successful
0812 09:45:47.378259 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":645,"thread_start_us":98,"threads_started":1}
I20260812 09:45:47.379096 25347 tablet_service.cc:1467] Processing CreateTablet for tablet 3b9f4604cc984ca284a7f2586229fef2 (DEFAULT_TABLE table=test-table [id=93402c0d19304ecf89f21b61786d2b4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 09:45:47.379611 25347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b9f4604cc984ca284a7f2586229fef2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:45:47.379633 24918 tablet_service.cc:1467] Processing CreateTablet for tablet 3b9f4604cc984ca284a7f2586229fef2 (DEFAULT_TABLE table=test-table [id=93402c0d19304ecf89f21b61786d2b4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 09:45:47.379895 24918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b9f4604cc984ca284a7f2586229fef2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:45:47.383389 25610 tablet_bootstrap.cc:492] T 3b9f4604cc984ca284a7f2586229fef2 P 22cbcc6238b04ef196d61a4623d51639: Bootstrap starting.
I20260812 09:45:47.384497 25610 tablet_bootstrap.cc:654] T 3b9f4604cc984ca284a7f2586229fef2 P 22cbcc6238b04ef196d61a4623d51639: Neither blocks nor log segments found. Creating new log.
I20260812 09:45:47.385252 25613 tablet_bootstrap.cc:492] T 3b9f4604cc984ca284a7f2586229fef2 P 433ef24880bd41ce9d2092d3748f7792: Bootstrap starting.
I20260812 09:45:47.385844 25610 tablet_bootstrap.cc:492] T 3b9f4604cc984ca284a7f2586229fef2 P 22cbcc6238b04ef196d61a4623d51639: No bootstrap required, opened a new log
I20260812 09:45:47.385931 25610 ts_tablet_manager.cc:1397] T 3b9f4604cc984ca284a7f2586229fef2 P 22cbcc6238b04ef196d61a4623d51639: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 09:45:47.386432 25610 raft_consensus.cc:348] T 3b9f4604cc984ca284a7f2586229fef2 P 22cbcc6238b04ef196d61a4623d51639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22cbcc6238b04ef196d61a4623d51639" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 33935 } } peers { permanent_uuid: "022e82db4dab49d29d2f320818a7f2f8" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 44373 } } peers { permanent_uuid: "433ef24880bd41ce9d2092d3748f7792" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 46293 } }
I20260812 09:45:47.386582 25610 raft_consensus.cc:374] T 3b9f4604cc984ca284a7f2586229fef2 P 22cbcc6238b04ef196d61a4623d51639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:45:47.386615 25610 raft_consensus.cc:729] T 3b9f4604cc984ca284a7f2586229fef2 P 22cbcc6238b04ef196d61a4623d51639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22cbcc6238b04ef196d61a4623d51639, State: Initialized, Role: FOLLOWER
I20260812 09:45:47.386740 25610 consensus_queue.cc:260] T 3b9f4604cc984ca284a7f2586229fef2 P 22cbcc6238b04ef196d61a4623d51639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22cbcc6238b04ef196d61a4623d51639" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 33935 } } peers { permanent_uuid: "022e82db4dab49d29d2f320818a7f2f8" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 44373 } } peers { permanent_uuid: "433ef24880bd41ce9d2092d3748f7792" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 46293 } }
I20260812 09:45:47.387090 25610 ts_tablet_manager.cc:1428] T 3b9f4604cc984ca284a7f2586229fef2 P 22cbcc6238b04ef196d61a4623d51639: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 09:45:47.387955 25586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:47.387697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:43093 (local address 127.13.99.1:40833)
0812 09:45:47.387830 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:47.387895 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:43093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260812 09:45:47.388686 25586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:47.388418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:43093 (local address 127.13.99.1:41263)
0812 09:45:47.388561 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:47.388613 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:43093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260812 09:45:47.390604 25025 heartbeater.cc:503] Master 127.13.99.61:36161 was elected leader, sending a full tablet report...
I20260812 09:45:47.391422 25613 tablet_bootstrap.cc:654] T 3b9f4604cc984ca284a7f2586229fef2 P 433ef24880bd41ce9d2092d3748f7792: Neither blocks nor log segments found. Creating new log.
I20260812 09:45:47.392896 25613 tablet_bootstrap.cc:492] T 3b9f4604cc984ca284a7f2586229fef2 P 433ef24880bd41ce9d2092d3748f7792: No bootstrap required, opened a new log
I20260812 09:45:47.392998 25613 ts_tablet_manager.cc:1397] T 3b9f4604cc984ca284a7f2586229fef2 P 433ef24880bd41ce9d2092d3748f7792: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260812 09:45:47.393463 25613 raft_consensus.cc:348] T 3b9f4604cc984ca284a7f2586229fef2 P 433ef24880bd41ce9d2092d3748f7792 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22cbcc6238b04ef196d61a4623d51639" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 33935 } } peers { permanent_uuid: "022e82db4dab49d29d2f320818a7f2f8" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 44373 } } peers { permanent_uuid: "433ef24880bd41ce9d2092d3748f7792" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 46293 } }
I20260812 09:45:47.393630 25613 raft_consensus.cc:374] T 3b9f4604cc984ca284a7f2586229fef2 P 433ef24880bd41ce9d2092d3748f7792 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:45:47.393648 25612 tablet_bootstrap.cc:492] T 3b9f4604cc984ca284a7f2586229fef2 P 022e82db4dab49d29d2f320818a7f2f8: Bootstrap starting.
I20260812 09:45:47.393702 25613 raft_consensus.cc:729] T 3b9f4604cc984ca284a7f2586229fef2 P 433ef24880bd41ce9d2092d3748f7792 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 433ef24880bd41ce9d2092d3748f7792, State: Initialized, Role: FOLLOWER
I20260812 09:45:47.393851 25613 consensus_queue.cc:260] T 3b9f4604cc984ca284a7f2586229fef2 P 433ef24880bd41ce9d2092d3748f7792 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22cbcc6238b04ef196d61a4623d51639" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 33935 } } peers { permanent_uuid: "022e82db4dab49d29d2f320818a7f2f8" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 44373 } } peers { permanent_uuid: "433ef24880bd41ce9d2092d3748f7792" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 46293 } }
I20260812 09:45:47.394271 25613 ts_tablet_manager.cc:1428] T 3b9f4604cc984ca284a7f2586229fef2 P 433ef24880bd41ce9d2092d3748f7792: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260812 09:45:47.394798 25612 tablet_bootstrap.cc:654] T 3b9f4604cc984ca284a7f2586229fef2 P 022e82db4dab49d29d2f320818a7f2f8: Neither blocks nor log segments found. Creating new log.
I20260812 09:45:47.395044 25583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:47.394531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:43093 (local address 127.13.99.2:38083)
0812 09:45:47.394902 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:47.394962 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:43093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":286}
I20260812 09:45:47.395869 25583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:47.395487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:43093 (local address 127.13.99.2:41915)
0812 09:45:47.395735 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:47.395792 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:43093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180}
I20260812 09:45:47.396129 25612 tablet_bootstrap.cc:492] T 3b9f4604cc984ca284a7f2586229fef2 P 022e82db4dab49d29d2f320818a7f2f8: No bootstrap required, opened a new log
I20260812 09:45:47.396226 25612 ts_tablet_manager.cc:1397] T 3b9f4604cc984ca284a7f2586229fef2 P 022e82db4dab49d29d2f320818a7f2f8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 09:45:47.396464 25235 heartbeater.cc:503] Master 127.13.99.61:36161 was elected leader, sending a full tablet report...
I20260812 09:45:47.396718 25612 raft_consensus.cc:348] T 3b9f4604cc984ca284a7f2586229fef2 P 022e82db4dab49d29d2f320818a7f2f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22cbcc6238b04ef196d61a4623d51639" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 33935 } } peers { permanent_uuid: "022e82db4dab49d29d2f320818a7f2f8" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 44373 } } peers { permanent_uuid: "433ef24880bd41ce9d2092d3748f7792" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 46293 } }
I20260812 09:45:47.396857 25612 raft_consensus.cc:374] T 3b9f4604cc984ca284a7f2586229fef2 P 022e82db4dab49d29d2f320818a7f2f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:45:47.396924 25612 raft_consensus.cc:729] T 3b9f4604cc984ca284a7f2586229fef2 P 022e82db4dab49d29d2f320818a7f2f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 022e82db4dab49d29d2f320818a7f2f8, State: Initialized, Role: FOLLOWER
I20260812 09:45:47.397074 25612 consensus_queue.cc:260] T 3b9f4604cc984ca284a7f2586229fef2 P 022e82db4dab49d29d2f320818a7f2f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22cbcc6238b04ef196d61a4623d51639" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 33935 } } peers { permanent_uuid: "022e82db4dab49d29d2f320818a7f2f8" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 44373 } } peers { permanent_uuid: "433ef24880bd41ce9d2092d3748f7792" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 46293 } }
I20260812 09:45:47.398940 25612 ts_tablet_manager.cc:1428] T 3b9f4604cc984ca284a7f2586229fef2 P 022e82db4dab49d29d2f320818a7f2f8: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260812 09:45:47.400663 25585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:47.400051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:43093 (local address 127.13.99.3:40585)
0812 09:45:47.400533 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:47.400591 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:43093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384}
I20260812 09:45:47.400823 25456 heartbeater.cc:503] Master 127.13.99.61:36161 was elected leader, sending a full tablet report...
I20260812 09:45:47.489496 25618 raft_consensus.cc:482] T 3b9f4604cc984ca284a7f2586229fef2 P 22cbcc6238b04ef196d61a4623d51639 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 09:45:47.489689 25618 raft_consensus.cc:504] T 3b9f4604cc984ca284a7f2586229fef2 P 22cbcc6238b04ef196d61a4623d51639 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22cbcc6238b04ef196d61a4623d51639" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 33935 } } peers { permanent_uuid: "022e82db4dab49d29d2f320818a7f2f8" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 44373 } } peers { permanent_uuid: "433ef24880bd41ce9d2092d3748f7792" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 46293 } }
I20260812 09:45:47.490159 25618 leader_election.cc:288] T 3b9f4604cc984ca284a7f2586229fef2 P 22cbcc6238b04ef196d61a4623d51639 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 022e82db4dab49d29d2f320818a7f2f8 (127.13.99.3:44373), 433ef24880bd41ce9d2092d3748f7792 (127.13.99.2:46293)
I20260812 09:45:47.490618 25378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b9f4604cc984ca284a7f2586229fef2" candidate_uuid: "22cbcc6238b04ef196d61a4623d51639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "022e82db4dab49d29d2f320818a7f2f8" is_pre_election: true
I20260812 09:45:47.490626 25164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b9f4604cc984ca284a7f2586229fef2" candidate_uuid: "22cbcc6238b04ef196d61a4623d51639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "433ef24880bd41ce9d2092d3748f7792" is_pre_election: true
I20260812 09:45:47.490749 25378 raft_consensus.cc:2454] T 3b9f4604cc984ca284a7f2586229fef2 P 022e82db4dab49d29d2f320818a7f2f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22cbcc6238b04ef196d61a4623d51639 in term 0.
I20260812 09:45:47.490824 25164 raft_consensus.cc:2454] T 3b9f4604cc984ca284a7f2586229fef2 P 433ef24880bd41ce9d2092d3748f7792 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22cbcc6238b04ef196d61a4623d51639 in term 0.
I20260812 09:45:47.491171 24851 leader_election.cc:302] T 3b9f4604cc984ca284a7f2586229fef2 P 22cbcc6238b04ef196d61a4623d51639 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 022e82db4dab49d29d2f320818a7f2f8, 22cbcc6238b04ef196d61a4623d51639; no voters:
I20260812 09:45:47.491326 25618 raft_consensus.cc:2781] T 3b9f4604cc984ca284a7f2586229fef2 P 22cbcc6238b04ef196d61a4623d51639 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 09:45:47.491387 25618 raft_consensus.cc:482] T 3b9f4604cc984ca284a7f2586229fef2 P 22cbcc6238b04ef196d61a4623d51639 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 09:45:47.491436 25618 raft_consensus.cc:3037] T 3b9f4604cc984ca284a7f2586229fef2 P 22cbcc6238b04ef196d61a4623d51639 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:45:47.494257 25618 raft_consensus.cc:504] T 3b9f4604cc984ca284a7f2586229fef2 P 22cbcc6238b04ef196d61a4623d51639 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22cbcc6238b04ef196d61a4623d51639" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 33935 } } peers { permanent_uuid: "022e82db4dab49d29d2f320818a7f2f8" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 44373 } } peers { permanent_uuid: "433ef24880bd41ce9d2092d3748f7792" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 46293 } }
I20260812 09:45:47.494623 25618 leader_election.cc:288] T 3b9f4604cc984ca284a7f2586229fef2 P 22cbcc6238b04ef196d61a4623d51639 [CANDIDATE]: Term 1 election: Requested vote from peers 022e82db4dab49d29d2f320818a7f2f8 (127.13.99.3:44373), 433ef24880bd41ce9d2092d3748f7792 (127.13.99.2:46293)
I20260812 09:45:47.495003 25378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b9f4604cc984ca284a7f2586229fef2" candidate_uuid: "22cbcc6238b04ef196d61a4623d51639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "022e82db4dab49d29d2f320818a7f2f8"
I20260812 09:45:47.495085 25164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b9f4604cc984ca284a7f2586229fef2" candidate_uuid: "22cbcc6238b04ef196d61a4623d51639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "433ef24880bd41ce9d2092d3748f7792"
I20260812 09:45:47.495144 25378 raft_consensus.cc:3037] T 3b9f4604cc984ca284a7f2586229fef2 P 022e82db4dab49d29d2f320818a7f2f8 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:45:47.495222 25164 raft_consensus.cc:3037] T 3b9f4604cc984ca284a7f2586229fef2 P 433ef24880bd41ce9d2092d3748f7792 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:45:47.496093 25378 raft_consensus.cc:2454] T 3b9f4604cc984ca284a7f2586229fef2 P 022e82db4dab49d29d2f320818a7f2f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22cbcc6238b04ef196d61a4623d51639 in term 1.
I20260812 09:45:47.496243 25164 raft_consensus.cc:2454] T 3b9f4604cc984ca284a7f2586229fef2 P 433ef24880bd41ce9d2092d3748f7792 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22cbcc6238b04ef196d61a4623d51639 in term 1.
I20260812 09:45:47.496430 24851 leader_election.cc:302] T 3b9f4604cc984ca284a7f2586229fef2 P 22cbcc6238b04ef196d61a4623d51639 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 022e82db4dab49d29d2f320818a7f2f8, 22cbcc6238b04ef196d61a4623d51639; no voters:
I20260812 09:45:47.496668 25618 raft_consensus.cc:2781] T 3b9f4604cc984ca284a7f2586229fef2 P 22cbcc6238b04ef196d61a4623d51639 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 09:45:47.496937 25618 raft_consensus.cc:686] T 3b9f4604cc984ca284a7f2586229fef2 P 22cbcc6238b04ef196d61a4623d51639 [term 1 LEADER]: Becoming Leader. State: Replica: 22cbcc6238b04ef196d61a4623d51639, State: Running, Role: LEADER
I20260812 09:45:47.497108 25618 consensus_queue.cc:237] T 3b9f4604cc984ca284a7f2586229fef2 P 22cbcc6238b04ef196d61a4623d51639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22cbcc6238b04ef196d61a4623d51639" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 33935 } } peers { permanent_uuid: "022e82db4dab49d29d2f320818a7f2f8" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 44373 } } peers { permanent_uuid: "433ef24880bd41ce9d2092d3748f7792" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 46293 } }
I20260812 09:45:47.497653 25586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:47.497401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:43093 (local address 127.13.99.1:52773)
0812 09:45:47.497509 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:47.497581 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:43093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260812 09:45:47.499022 24478 catalog_manager.cc:5495] T 3b9f4604cc984ca284a7f2586229fef2 P 22cbcc6238b04ef196d61a4623d51639 reported cstate change: term changed from 0 to 1, leader changed from <none> to 22cbcc6238b04ef196d61a4623d51639 (127.13.99.1). New cstate: current_term: 1 leader_uuid: "22cbcc6238b04ef196d61a4623d51639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22cbcc6238b04ef196d61a4623d51639" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 33935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "022e82db4dab49d29d2f320818a7f2f8" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 44373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "433ef24880bd41ce9d2092d3748f7792" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 46293 } health_report { overall_health: UNKNOWN } } }
I20260812 09:45:47.512619 25603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:47.509544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56552 (local address 127.13.99.1:33935)
0812 09:45:47.509665 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:47.509669 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:47.509681 (+ 12us) server_negotiation.cc:407] Connection header received
0812 09:45:47.509762 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:47.509764 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:47.509804 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:47.509867 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 09:45:47.510091 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:47.510586 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:47.511486 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:47.511726 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:47.511839 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:47.512107 (+ 268us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 09:45:47.512178 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 09:45:47.512258 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:47.512389 (+ 131us) server_negotiation.cc:299] Negotiation successful
0812 09:45:47.512442 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260812 09:45:47.514500 25378 raft_consensus.cc:1261] T 3b9f4604cc984ca284a7f2586229fef2 P 022e82db4dab49d29d2f320818a7f2f8 [term 1 FOLLOWER]: Refusing update from remote peer 22cbcc6238b04ef196d61a4623d51639: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 09:45:47.514533 25164 raft_consensus.cc:1261] T 3b9f4604cc984ca284a7f2586229fef2 P 433ef24880bd41ce9d2092d3748f7792 [term 1 FOLLOWER]: Refusing update from remote peer 22cbcc6238b04ef196d61a4623d51639: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 09:45:47.514971 25629 consensus_queue.cc:1035] T 3b9f4604cc984ca284a7f2586229fef2 P 22cbcc6238b04ef196d61a4623d51639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "022e82db4dab49d29d2f320818a7f2f8" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 44373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 09:45:47.515141 25618 consensus_queue.cc:1035] T 3b9f4604cc984ca284a7f2586229fef2 P 22cbcc6238b04ef196d61a4623d51639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "433ef24880bd41ce9d2092d3748f7792" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 46293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 09:45:47.516896 25583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:47.516593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:43093 (local address 127.13.99.2:38749)
0812 09:45:47.516748 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:47.516806 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:43093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260812 09:45:47.518221 25585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:47.517682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:43093 (local address 127.13.99.3:53091)
0812 09:45:47.518063 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:47.518120 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:43093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":305}
I20260812 09:45:47.520182 25583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:47.519847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:43093 (local address 127.13.99.2:47725)
0812 09:45:47.520051 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:47.520105 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:43093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
I20260812 09:45:47.524358 25586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:47.524013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:43093 (local address 127.13.99.1:50175)
0812 09:45:47.524240 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:47.524301 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:43093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151}
I20260812 09:45:47.524770 24479 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56032:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 09:45:47.524945 24479 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56032:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 09:45:47.525810 25586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:47.525584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:43093 (local address 127.13.99.1:39141)
0812 09:45:47.525679 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:47.525730 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:43093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260812 09:45:47.526221 25585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:47.526011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:43093 (local address 127.13.99.3:38995)
0812 09:45:47.526106 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:47.526157 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:43093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260812 09:45:47.529091 24479 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cf7b481d137c42268998cd7a08f2d65e: Sending DeleteTablet for 3 replicas of tablet 3b9f4604cc984ca284a7f2586229fef2
I20260812 09:45:47.529728 24918 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b9f4604cc984ca284a7f2586229fef2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 09:45:47 UTC) from {username='oryx', principal='oryx/127.13.99.61@KRBTEST.COM'} at 127.0.0.1:56536
I20260812 09:45:47.529811 25347 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b9f4604cc984ca284a7f2586229fef2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 09:45:47 UTC) from {username='oryx', principal='oryx/127.13.99.61@KRBTEST.COM'} at 127.0.0.1:47166
I20260812 09:45:47.530076 25650 tablet_replica.cc:330] T 3b9f4604cc984ca284a7f2586229fef2 P 22cbcc6238b04ef196d61a4623d51639: stopping tablet replica
I20260812 09:45:47.530055 25134 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b9f4604cc984ca284a7f2586229fef2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 09:45:47 UTC) from {username='oryx', principal='oryx/127.13.99.61@KRBTEST.COM'} at 127.0.0.1:59100
I20260812 09:45:47.530145 25653 tablet_replica.cc:330] T 3b9f4604cc984ca284a7f2586229fef2 P 022e82db4dab49d29d2f320818a7f2f8: stopping tablet replica
I20260812 09:45:47.530292 25650 raft_consensus.cc:2229] T 3b9f4604cc984ca284a7f2586229fef2 P 22cbcc6238b04ef196d61a4623d51639 [term 1 LEADER]: Raft consensus shutting down.
I20260812 09:45:47.530309 25653 raft_consensus.cc:2229] T 3b9f4604cc984ca284a7f2586229fef2 P 022e82db4dab49d29d2f320818a7f2f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 09:45:47.530365 25654 tablet_replica.cc:330] T 3b9f4604cc984ca284a7f2586229fef2 P 433ef24880bd41ce9d2092d3748f7792: stopping tablet replica
I20260812 09:45:47.530445 25653 raft_consensus.cc:2258] T 3b9f4604cc984ca284a7f2586229fef2 P 022e82db4dab49d29d2f320818a7f2f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 09:45:47.530539 25650 raft_consensus.cc:2258] T 3b9f4604cc984ca284a7f2586229fef2 P 22cbcc6238b04ef196d61a4623d51639 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 09:45:47.530543 25654 raft_consensus.cc:2229] T 3b9f4604cc984ca284a7f2586229fef2 P 433ef24880bd41ce9d2092d3748f7792 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 09:45:47.530937 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 24683
I20260812 09:45:47.541550 25653 ts_tablet_manager.cc:1905] T 3b9f4604cc984ca284a7f2586229fef2 P 022e82db4dab49d29d2f320818a7f2f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 09:45:47.530670 25654 raft_consensus.cc:2258] T 3b9f4604cc984ca284a7f2586229fef2 P 433ef24880bd41ce9d2092d3748f7792 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 09:45:47.545506 25653 ts_tablet_manager.cc:1918] T 3b9f4604cc984ca284a7f2586229fef2 P 022e82db4dab49d29d2f320818a7f2f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 09:45:47.545614 25653 log.cc:1196] T 3b9f4604cc984ca284a7f2586229fef2 P 022e82db4dab49d29d2f320818a7f2f8: Deleting WAL directory at /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/ts-2/wal/wals/3b9f4604cc984ca284a7f2586229fef2
I20260812 09:45:47.546010 25653 ts_tablet_manager.cc:1939] T 3b9f4604cc984ca284a7f2586229fef2 P 022e82db4dab49d29d2f320818a7f2f8: Deleting consensus metadata
I20260812 09:45:47.547053 24458 catalog_manager.cc:4841] TS 022e82db4dab49d29d2f320818a7f2f8 (127.13.99.3:44373): tablet 3b9f4604cc984ca284a7f2586229fef2 (table test-table [id=93402c0d19304ecf89f21b61786d2b4b]) successfully deleted
I20260812 09:45:47.552229 25654 ts_tablet_manager.cc:1905] T 3b9f4604cc984ca284a7f2586229fef2 P 433ef24880bd41ce9d2092d3748f7792: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 09:45:47.554268 25654 ts_tablet_manager.cc:1918] T 3b9f4604cc984ca284a7f2586229fef2 P 433ef24880bd41ce9d2092d3748f7792: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 09:45:47.554395 25654 log.cc:1196] T 3b9f4604cc984ca284a7f2586229fef2 P 433ef24880bd41ce9d2092d3748f7792: Deleting WAL directory at /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786527905902985-13708-0/minicluster-data/ts-1/wal/wals/3b9f4604cc984ca284a7f2586229fef2
I20260812 09:45:47.554683 25654 ts_tablet_manager.cc:1939] T 3b9f4604cc984ca284a7f2586229fef2 P 433ef24880bd41ce9d2092d3748f7792: Deleting consensus metadata
I20260812 09:45:47.555409 24457 catalog_manager.cc:4841] TS 433ef24880bd41ce9d2092d3748f7792 (127.13.99.2:46293): tablet 3b9f4604cc984ca284a7f2586229fef2 (table test-table [id=93402c0d19304ecf89f21b61786d2b4b]) successfully deleted
I20260812 09:45:47.560385 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 25033
I20260812 09:45:47.560648 25605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:47.560290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.1:33935 (local address 127.0.0.1:56564)
0812 09:45:47.560438 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:47.560549 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.1:33935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260812 09:45:47.569443 24456 catalog_manager.cc:4584] TS 22cbcc6238b04ef196d61a4623d51639 (127.13.99.1:33935): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3b9f4604cc984ca284a7f2586229fef2: Network error: Client connection negotiation failed: client connection to 127.13.99.1:33935: connect: Connection refused (error 111)
I20260812 09:45:47.575685 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 25244
I20260812 09:45:47.595278 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 24274
I20260812 09:45:47.607692 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 24431
2026-08-12T09:45:47Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7803 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 09:45:47 dist-test-slave-jq4t krb5kdc[25661](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 09:45:47 dist-test-slave-jq4t krb5kdc[25661](info): set up 2 sockets
Aug 12 09:45:47 dist-test-slave-jq4t krb5kdc[25661](info): commencing operation
krb5kdc: starting...
W20260812 09:45:48.880414 13708 mini_kdc.cc:121] Time spent starting KDC: real 1.242s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/krb5kdc/test-user.keytab.
Aug 12 09:45:48 dist-test-slave-jq4t krb5kdc[25661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527948, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T09:45:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T09:45:48Z Disabled control of system clock
WARNING: no policy specified for oryx/127.13.99.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.99.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.99.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/krb5kdc/oryx_127.13.99.62.keytab.
Entry for principal oryx/127.13.99.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/krb5kdc/oryx_127.13.99.62.keytab.
I20260812 09:45:49.012943 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.62:35893
--webserver_interface=127.13.99.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/krb5kdc/oryx_127.13.99.62.keytab
--principal=oryx/127.13.99.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:44777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.99.62:35893
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:45:49.127326 25677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:45:49.127656 25677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:45:49.127741 25677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:45:49.131785 25677 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 09:45:49.131877 25677 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260812 09:45:49.131910 25677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:45:49.131951 25677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:45:49.132010 25677 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 09:45:49.132050 25677 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 09:45:49.137214 25677 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:44777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.99.62:35893
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/krb5kdc/oryx_127.13.99.62.keytab
--principal=oryx/127.13.99.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.62:35893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.99.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25677
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:45:49.138252 25677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:45:49.139284 25677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 09:45:49.148226 25677 server_base.cc:1034] running on GCE node
W20260812 09:45:49.148257 25688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:49.148257 25683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:49.148454 25682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:45:49.149065 25677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:45:49.150053 25677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:45:49.151216 25677 hybrid_clock.cc:648] HybridClock initialized: now 1786527949151195 us; error 33 us; skew 500 ppm
Aug 12 09:45:49 dist-test-slave-jq4t krb5kdc[25661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527949, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.99.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:45:49.158142 25677 init.cc:377] Logged in from keytab as oryx/127.13.99.62@KRBTEST.COM (short username oryx)
I20260812 09:45:49.159147 25677 webserver.cc:460] Webserver started at http://127.13.99.62:39385/ using document root <none> and password file <none>
I20260812 09:45:49.159745 25677 fs_manager.cc:359] Metadata directory not provided
I20260812 09:45:49.159847 25677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:45:49.160069 25677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:45:49.162227 25677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/minicluster-data/master-0/data/instance:
uuid: "7fb7f7ee6bd04ed7bb41ec750ae6d72a"
format_stamp: "Formatted at 2026-08-12 09:45:49 on dist-test-slave-jq4t"
server_key: "7f7a980d4fc19cf8e20f0ae292f7551d"
server_key_iv: "d720bd634c83b68eb4ed0e3406262bfa"
server_key_version: "encryptionkey@0"
I20260812 09:45:49.162828 25677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/minicluster-data/master-0/wal/instance:
uuid: "7fb7f7ee6bd04ed7bb41ec750ae6d72a"
format_stamp: "Formatted at 2026-08-12 09:45:49 on dist-test-slave-jq4t"
server_key: "7f7a980d4fc19cf8e20f0ae292f7551d"
server_key_iv: "d720bd634c83b68eb4ed0e3406262bfa"
server_key_version: "encryptionkey@0"
I20260812 09:45:49.166635 25677 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.005s
I20260812 09:45:49.169287 25699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:49.170529 25677 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260812 09:45:49.170694 25677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/minicluster-data/master-0/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/minicluster-data/master-0/wal
uuid: "7fb7f7ee6bd04ed7bb41ec750ae6d72a"
format_stamp: "Formatted at 2026-08-12 09:45:49 on dist-test-slave-jq4t"
server_key: "7f7a980d4fc19cf8e20f0ae292f7551d"
server_key_iv: "d720bd634c83b68eb4ed0e3406262bfa"
server_key_version: "encryptionkey@0"
I20260812 09:45:49.170826 25677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:45:49.186215 25677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:45:49.189030 25677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:45:49.189234 25677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:45:49.197530 25677 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.62:35893
I20260812 09:45:49.197541 25797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.62:35893 every 8 connection(s)
I20260812 09:45:49.198469 25677 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
I20260812 09:45:49.200640 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 25677
I20260812 09:45:49.200764 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/minicluster-data/master-0/wal/instance
I20260812 09:45:49.201093 13708 external_mini_cluster.cc:1428] Setting key 5550b22765ebb6d2c82520c8b8dd7f37
I20260812 09:45:49.202035 25799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:45:49.208666 25799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fb7f7ee6bd04ed7bb41ec750ae6d72a: Bootstrap starting.
Aug 12 09:45:49 dist-test-slave-jq4t krb5kdc[25661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527948, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM
I20260812 09:45:49.211295 25799 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7fb7f7ee6bd04ed7bb41ec750ae6d72a: Neither blocks nor log segments found. Creating new log.
I20260812 09:45:49.212373 25799 log.cc:824] T 00000000000000000000000000000000 P 7fb7f7ee6bd04ed7bb41ec750ae6d72a: Log is configured to *not* fsync() on all Append() calls
I20260812 09:45:49.214531 25799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fb7f7ee6bd04ed7bb41ec750ae6d72a: No bootstrap required, opened a new log
I20260812 09:45:49.216372 25803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:49.202471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55106 (local address 127.13.99.62:35893)
0812 09:45:49.202897 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:49.202916 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:49.202966 (+ 50us) server_negotiation.cc:407] Connection header received
0812 09:45:49.203602 (+ 636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:49.203619 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:49.203947 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:49.204226 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:49.205225 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:49.206263 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:49.207512 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:49.208117 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:49.208255 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:49.211237 (+ 2982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:49.211243 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:49.211259 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:49.211288 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:49.213420 (+ 2132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:49.213892 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:49.213899 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:49.213904 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:49.214031 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:49.214353 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:49.214356 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:49.214358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:49.214837 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:49.215070 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:49.215384 (+ 314us) server_negotiation.cc:299] Negotiation successful
0812 09:45:49.215677 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":143,"threads_started":1}
I20260812 09:45:49.218086 25799 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7fb7f7ee6bd04ed7bb41ec750ae6d72a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fb7f7ee6bd04ed7bb41ec750ae6d72a" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 35893 } }
I20260812 09:45:49.218397 25799 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7fb7f7ee6bd04ed7bb41ec750ae6d72a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:45:49.218530 25799 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7fb7f7ee6bd04ed7bb41ec750ae6d72a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fb7f7ee6bd04ed7bb41ec750ae6d72a, State: Initialized, Role: FOLLOWER
I20260812 09:45:49.219027 25799 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7fb7f7ee6bd04ed7bb41ec750ae6d72a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fb7f7ee6bd04ed7bb41ec750ae6d72a" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 35893 } }
I20260812 09:45:49.219164 25799 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7fb7f7ee6bd04ed7bb41ec750ae6d72a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 09:45:49.219254 25799 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7fb7f7ee6bd04ed7bb41ec750ae6d72a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 09:45:49.219373 25799 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7fb7f7ee6bd04ed7bb41ec750ae6d72a [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:45:49.220260 25799 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7fb7f7ee6bd04ed7bb41ec750ae6d72a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fb7f7ee6bd04ed7bb41ec750ae6d72a" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 35893 } }
I20260812 09:45:49.220654 25799 leader_election.cc:302] T 00000000000000000000000000000000 P 7fb7f7ee6bd04ed7bb41ec750ae6d72a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7fb7f7ee6bd04ed7bb41ec750ae6d72a; no voters:
I20260812 09:45:49.220963 25799 leader_election.cc:288] T 00000000000000000000000000000000 P 7fb7f7ee6bd04ed7bb41ec750ae6d72a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 09:45:49.221117 25806 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7fb7f7ee6bd04ed7bb41ec750ae6d72a [term 1 FOLLOWER]: Leader election won for term 1
I20260812 09:45:49.221354 25806 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7fb7f7ee6bd04ed7bb41ec750ae6d72a [term 1 LEADER]: Becoming Leader. State: Replica: 7fb7f7ee6bd04ed7bb41ec750ae6d72a, State: Running, Role: LEADER
I20260812 09:45:49.221702 25806 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7fb7f7ee6bd04ed7bb41ec750ae6d72a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fb7f7ee6bd04ed7bb41ec750ae6d72a" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 35893 } }
I20260812 09:45:49.222189 25799 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7fb7f7ee6bd04ed7bb41ec750ae6d72a [sys.catalog]: configured and running, proceeding with master startup.
I20260812 09:45:49.223595 25810 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fb7f7ee6bd04ed7bb41ec750ae6d72a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7fb7f7ee6bd04ed7bb41ec750ae6d72a. Latest consensus state: current_term: 1 leader_uuid: "7fb7f7ee6bd04ed7bb41ec750ae6d72a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fb7f7ee6bd04ed7bb41ec750ae6d72a" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 35893 } } }
I20260812 09:45:49.223652 25808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fb7f7ee6bd04ed7bb41ec750ae6d72a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7fb7f7ee6bd04ed7bb41ec750ae6d72a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fb7f7ee6bd04ed7bb41ec750ae6d72a" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 35893 } } }
I20260812 09:45:49.223726 25810 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fb7f7ee6bd04ed7bb41ec750ae6d72a [sys.catalog]: This master's current role is: LEADER
I20260812 09:45:49.223744 25808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fb7f7ee6bd04ed7bb41ec750ae6d72a [sys.catalog]: This master's current role is: LEADER
I20260812 09:45:49.224160 25820 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 09:45:49.227355 25820 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 09:45:49.233357 25820 catalog_manager.cc:1347] Generated new cluster ID: d7e58f19dc7246aeb3092d3545bec662
I20260812 09:45:49.233423 25820 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 09:45:49.242184 25820 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 09:45:49.243053 25820 catalog_manager.cc:1504] Loading token signing keys...
I20260812 09:45:49.249673 25820 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7fb7f7ee6bd04ed7bb41ec750ae6d72a: Generated new TSK 0
I20260812 09:45:49.250355 25820 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.13.99.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.99.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.99.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/krb5kdc/oryx_127.13.99.1.keytab.
Entry for principal oryx/127.13.99.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/krb5kdc/oryx_127.13.99.1.keytab.
I20260812 09:45:49.287786 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.99.1:0
--local_ip_for_outbound_sockets=127.13.99.1
--webserver_interface=127.13.99.1
--webserver_port=0
--tserver_master_addrs=127.13.99.62:35893
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/krb5kdc/oryx_127.13.99.1.keytab
--principal=oryx/127.13.99.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:44777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:45:49.424520 25840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:45:49.424753 25840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:45:49.424795 25840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:45:49.428669 25840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:45:49.428721 25840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:45:49.428757 25840 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 09:45:49.428788 25840 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.99.1
I20260812 09:45:49.433790 25840 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:44777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/krb5kdc/oryx_127.13.99.1.keytab
--principal=oryx/127.13.99.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.99.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.99.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.99.62:35893
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25840
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.99.1
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:45:49.434733 25840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:45:49.435683 25840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:45:49.444806 25846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:49.444936 25850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:49.444821 25848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:45:49.445362 25840 server_base.cc:1034] running on GCE node
I20260812 09:45:49.445793 25840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:45:49.446424 25840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:45:49.447619 25840 hybrid_clock.cc:648] HybridClock initialized: now 1786527949447604 us; error 26 us; skew 500 ppm
Aug 12 09:45:49 dist-test-slave-jq4t krb5kdc[25661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527949, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.99.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:45:49.450421 25840 init.cc:377] Logged in from keytab as oryx/127.13.99.1@KRBTEST.COM (short username oryx)
I20260812 09:45:49.451282 25840 webserver.cc:460] Webserver started at http://127.13.99.1:44509/ using document root <none> and password file <none>
I20260812 09:45:49.451857 25840 fs_manager.cc:359] Metadata directory not provided
I20260812 09:45:49.451951 25840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:45:49.452185 25840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:45:49.454258 25840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/minicluster-data/ts-0/data/instance:
uuid: "c0742b25a62b4793bf20353f0d1f2a95"
format_stamp: "Formatted at 2026-08-12 09:45:49 on dist-test-slave-jq4t"
server_key: "f3b24b67b28e87fb4b5439fb978dba54"
server_key_iv: "e4e89fb57457997af5dcdac81dee04ae"
server_key_version: "encryptionkey@0"
I20260812 09:45:49.454898 25840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/minicluster-data/ts-0/wal/instance:
uuid: "c0742b25a62b4793bf20353f0d1f2a95"
format_stamp: "Formatted at 2026-08-12 09:45:49 on dist-test-slave-jq4t"
server_key: "f3b24b67b28e87fb4b5439fb978dba54"
server_key_iv: "e4e89fb57457997af5dcdac81dee04ae"
server_key_version: "encryptionkey@0"
I20260812 09:45:49.458760 25840 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 09:45:49.461362 25861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:49.462620 25840 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 09:45:49.462774 25840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/minicluster-data/ts-0/wal
uuid: "c0742b25a62b4793bf20353f0d1f2a95"
format_stamp: "Formatted at 2026-08-12 09:45:49 on dist-test-slave-jq4t"
server_key: "f3b24b67b28e87fb4b5439fb978dba54"
server_key_iv: "e4e89fb57457997af5dcdac81dee04ae"
server_key_version: "encryptionkey@0"
I20260812 09:45:49.462888 25840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:45:49.476090 25840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:45:49.478842 25840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:45:49.479045 25840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:45:49.480439 25840 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:45:49.480535 25840 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:49.480628 25840 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:45:49.480683 25840 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 09:45:49 dist-test-slave-jq4t krb5kdc[25661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527949, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.99.1@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM
I20260812 09:45:49.491868 25840 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.1:35609
I20260812 09:45:49.491983 26030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.1:35609 every 8 connection(s)
I20260812 09:45:49.492875 25840 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/minicluster-data/ts-0/data/info.pb
I20260812 09:45:49.494571 25803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:49.481300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.1:33649 (local address 127.13.99.62:35893)
0812 09:45:49.481437 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:49.481441 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:49.482472 (+ 1031us) server_negotiation.cc:407] Connection header received
0812 09:45:49.483403 (+ 931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:49.483406 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:49.483469 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:49.483558 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:49.485363 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:49.485985 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:49.487145 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:49.487403 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:49.487514 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:49.490764 (+ 3250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:49.490768 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:49.490771 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:49.490813 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:49.492391 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:49.492918 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:49.492923 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:49.492925 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:49.493038 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:49.493426 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:49.493430 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:49.493431 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:49.493679 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:49.493789 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:49.494328 (+ 539us) server_negotiation.cc:299] Negotiation successful
0812 09:45:49.494451 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260812 09:45:49.495157 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 25840
I20260812 09:45:49.495296 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/minicluster-data/ts-0/wal/instance
I20260812 09:45:49.495410 25874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:49.481825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:35893 (local address 127.13.99.1:33649)
0812 09:45:49.482327 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:49.482367 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:49.483195 (+ 828us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:49.483893 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:49.483904 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:49.484437 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:45:49.485185 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:49.485209 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:49.486125 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:49.486128 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:49.487002 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:49.487011 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:49.487598 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:49.487604 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:49.487807 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:49.488434 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:45:49.488453 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:45:49.490600 (+ 2147us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:45:49.492545 (+ 1945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:49.492551 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:49.492563 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:49.492809 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:49.493140 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:49.493144 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:49.493146 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:49.493286 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:49.493794 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:45:49.493799 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:45:49.494094 (+ 295us) client_negotiation.cc:769] Sending connection context
0812 09:45:49.494241 (+ 147us) client_negotiation.cc:240] Negotiation successful
0812 09:45:49.494467 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":143,"threads_started":1}
I20260812 09:45:49.495656 13708 external_mini_cluster.cc:1428] Setting key d998614d98a4add1617e13d1bda7907e
I20260812 09:45:49.497154 26032 heartbeater.cc:347] Connected to a master server at 127.13.99.62:35893
I20260812 09:45:49.497313 26032 heartbeater.cc:464] Registering TS with master...
I20260812 09:45:49.497886 26032 heartbeater.cc:511] Master 127.13.99.62:35893 requested a full tablet report, sending...
I20260812 09:45:49.499605 25725 ts_manager.cc:194] Registered new tserver with Master: c0742b25a62b4793bf20353f0d1f2a95 (127.13.99.1:35609)
I20260812 09:45:49.501161 25725 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.99.1@KRBTEST.COM'} at 127.13.99.1:33649
I20260812 09:45:49.507632 25803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:49.499227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.1:43403 (local address 127.13.99.62:35893)
0812 09:45:49.499334 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:49.499338 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:49.499564 (+ 226us) server_negotiation.cc:407] Connection header received
0812 09:45:49.499631 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:49.499634 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:49.499681 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:49.499783 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:49.500224 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:49.501033 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:49.502257 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:49.502645 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:49.502767 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:49.504266 (+ 1499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:49.504269 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:49.504271 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:49.504293 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:49.505871 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:49.506432 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:49.506435 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:49.506436 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:49.506526 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:49.506849 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:49.506852 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:49.506853 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:49.507021 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:49.507101 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:49.507419 (+ 318us) server_negotiation.cc:299] Negotiation successful
0812 09:45:49.507518 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260812 09:45:49.507768 26039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:49.499187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:35893 (local address 127.13.99.1:43403)
0812 09:45:49.499475 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:49.499487 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:49.499565 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:49.499842 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:49.499844 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:49.500002 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:45:49.500109 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:49.500114 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:49.501174 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:49.501179 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:49.502132 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:49.502142 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:49.502907 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:49.502913 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:49.503061 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:49.503579 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:45:49.503599 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:45:49.504103 (+ 504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:45:49.505989 (+ 1886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:49.505994 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:49.505999 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:49.506301 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:49.506639 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:49.506643 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:49.506645 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:49.506748 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:49.507123 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:45:49.507126 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:45:49.507240 (+ 114us) client_negotiation.cc:769] Sending connection context
0812 09:45:49.507361 (+ 121us) client_negotiation.cc:240] Negotiation successful
0812 09:45:49.507524 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"mutex_wait_us":11,"thread_start_us":89,"threads_started":1}
WARNING: no policy specified for oryx/127.13.99.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.99.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.99.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/krb5kdc/oryx_127.13.99.2.keytab.
Entry for principal oryx/127.13.99.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/krb5kdc/oryx_127.13.99.2.keytab.
I20260812 09:45:49.529645 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.99.2:0
--local_ip_for_outbound_sockets=127.13.99.2
--webserver_interface=127.13.99.2
--webserver_port=0
--tserver_master_addrs=127.13.99.62:35893
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/krb5kdc/oryx_127.13.99.2.keytab
--principal=oryx/127.13.99.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:44777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:45:49.648983 26044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:45:49.649236 26044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:45:49.649281 26044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:45:49.653290 26044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:45:49.653348 26044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:45:49.653388 26044 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 09:45:49.653419 26044 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.99.2
I20260812 09:45:49.658690 26044 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:44777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/krb5kdc/oryx_127.13.99.2.keytab
--principal=oryx/127.13.99.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.99.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.99.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.99.62:35893
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26044
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.99.2
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:45:49.659701 26044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:45:49.660756 26044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:45:49.670574 26050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:49.670588 26055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:45:49.670596 26044 server_base.cc:1034] running on GCE node
W20260812 09:45:49.670837 26051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:45:49.671264 26044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:45:49.671891 26044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:45:49.673163 26044 hybrid_clock.cc:648] HybridClock initialized: now 1786527949673125 us; error 48 us; skew 500 ppm
Aug 12 09:45:49 dist-test-slave-jq4t krb5kdc[25661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527949, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.99.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:45:49.676070 26044 init.cc:377] Logged in from keytab as oryx/127.13.99.2@KRBTEST.COM (short username oryx)
I20260812 09:45:49.677021 26044 webserver.cc:460] Webserver started at http://127.13.99.2:38635/ using document root <none> and password file <none>
I20260812 09:45:49.677585 26044 fs_manager.cc:359] Metadata directory not provided
I20260812 09:45:49.677677 26044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:45:49.677922 26044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:45:49.679965 26044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/minicluster-data/ts-1/data/instance:
uuid: "49fa8338699749498ff4283df5548ab3"
format_stamp: "Formatted at 2026-08-12 09:45:49 on dist-test-slave-jq4t"
server_key: "f672eb9c01643227741311f57bfe2373"
server_key_iv: "a18068a43af2e3f1f65c3570c03aa7ec"
server_key_version: "encryptionkey@0"
I20260812 09:45:49.680603 26044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/minicluster-data/ts-1/wal/instance:
uuid: "49fa8338699749498ff4283df5548ab3"
format_stamp: "Formatted at 2026-08-12 09:45:49 on dist-test-slave-jq4t"
server_key: "f672eb9c01643227741311f57bfe2373"
server_key_iv: "a18068a43af2e3f1f65c3570c03aa7ec"
server_key_version: "encryptionkey@0"
I20260812 09:45:49.684396 26044 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260812 09:45:49.686861 26066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:49.688094 26044 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 09:45:49.688244 26044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/minicluster-data/ts-1/wal
uuid: "49fa8338699749498ff4283df5548ab3"
format_stamp: "Formatted at 2026-08-12 09:45:49 on dist-test-slave-jq4t"
server_key: "f672eb9c01643227741311f57bfe2373"
server_key_iv: "a18068a43af2e3f1f65c3570c03aa7ec"
server_key_version: "encryptionkey@0"
I20260812 09:45:49.688360 26044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:45:49.706837 26044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:45:49.710067 26044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:45:49.710279 26044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:45:49.711822 26044 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:45:49.711898 26044 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:49.712007 26044 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:45:49.712064 26044 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 09:45:49 dist-test-slave-jq4t krb5kdc[25661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527949, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.99.2@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM
I20260812 09:45:49.723367 26044 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.2:36055
I20260812 09:45:49.724397 26044 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/minicluster-data/ts-1/data/info.pb
I20260812 09:45:49.725275 26246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.2:36055 every 8 connection(s)
I20260812 09:45:49.725903 25803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:49.712510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.2:55311 (local address 127.13.99.62:35893)
0812 09:45:49.712642 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:49.712647 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:49.713601 (+ 954us) server_negotiation.cc:407] Connection header received
0812 09:45:49.714536 (+ 935us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:49.714539 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:49.714596 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:49.714706 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:49.716180 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:49.717120 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:49.718361 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:49.718605 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:49.718724 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:49.721707 (+ 2983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:49.721718 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:49.721721 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:49.721747 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:49.723532 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:49.724141 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:49.724145 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:49.724147 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:49.724236 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:49.724690 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:49.724694 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:49.724696 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:49.724948 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:49.725048 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:49.725671 (+ 623us) server_negotiation.cc:299] Negotiation successful
0812 09:45:49.725782 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260812 09:45:49.726670 26080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:49.712963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:35893 (local address 127.13.99.2:55311)
0812 09:45:49.713443 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:49.713492 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:49.714321 (+ 829us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:49.714891 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:49.714901 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:49.715362 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:45:49.716006 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:49.716035 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:49.717271 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:49.717276 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:49.718220 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:49.718230 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:49.718793 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:49.718797 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:49.718953 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:49.719539 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:45:49.719555 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:45:49.721538 (+ 1983us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:45:49.723654 (+ 2116us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:49.723660 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:49.723675 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:49.724047 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:49.724369 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:49.724374 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:49.724380 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:49.724573 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:49.725104 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:45:49.725110 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:45:49.725393 (+ 283us) client_negotiation.cc:769] Sending connection context
0812 09:45:49.725572 (+ 179us) client_negotiation.cc:240] Negotiation successful
0812 09:45:49.725815 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":133,"threads_started":1}
I20260812 09:45:49.727293 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 26044
I20260812 09:45:49.727469 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/minicluster-data/ts-1/wal/instance
I20260812 09:45:49.727788 13708 external_mini_cluster.cc:1428] Setting key dc58c1b62b4e180d5e393bdf51d40959
I20260812 09:45:49.728042 26247 heartbeater.cc:347] Connected to a master server at 127.13.99.62:35893
I20260812 09:45:49.728199 26247 heartbeater.cc:464] Registering TS with master...
I20260812 09:45:49.728809 26247 heartbeater.cc:511] Master 127.13.99.62:35893 requested a full tablet report, sending...
I20260812 09:45:49.729895 25720 ts_manager.cc:194] Registered new tserver with Master: 49fa8338699749498ff4283df5548ab3 (127.13.99.2:36055)
I20260812 09:45:49.730626 25720 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.99.2@KRBTEST.COM'} at 127.13.99.2:55311
I20260812 09:45:49.737993 25803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:49.730751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.2:55327 (local address 127.13.99.62:35893)
0812 09:45:49.730877 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:49.730883 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:49.730899 (+ 16us) server_negotiation.cc:407] Connection header received
0812 09:45:49.731060 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:49.731066 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:49.731119 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:49.731173 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:49.731914 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:49.732422 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:49.733297 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:49.733532 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:49.733641 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:49.734698 (+ 1057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:49.734702 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:49.734704 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:49.734728 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:49.736159 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:49.736669 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:49.736673 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:49.736675 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:49.736763 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:49.737060 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:49.737065 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:49.737067 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:49.737281 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:49.737401 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:49.737731 (+ 330us) server_negotiation.cc:299] Negotiation successful
0812 09:45:49.737843 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260812 09:45:49.738006 26253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:49.730504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:35893 (local address 127.13.99.2:55327)
0812 09:45:49.730814 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:49.730831 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:49.730927 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:49.731342 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:49.731348 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:49.731605 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:45:49.731758 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:49.731769 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:49.732626 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:49.732629 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:49.733178 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:49.733184 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:49.733646 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:49.733650 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:49.733738 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:49.734260 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:45:49.734272 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:45:49.734566 (+ 294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:45:49.736286 (+ 1720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:49.736291 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:49.736293 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:49.736559 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:49.736863 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:49.736867 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:49.736869 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:49.736961 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:49.737393 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:45:49.737397 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:45:49.737540 (+ 143us) client_negotiation.cc:769] Sending connection context
0812 09:45:49.737671 (+ 131us) client_negotiation.cc:240] Negotiation successful
0812 09:45:49.737838 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":86,"threads_started":1}
WARNING: no policy specified for oryx/127.13.99.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.99.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.99.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/krb5kdc/oryx_127.13.99.3.keytab.
Entry for principal oryx/127.13.99.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/krb5kdc/oryx_127.13.99.3.keytab.
I20260812 09:45:49.759117 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.99.3:0
--local_ip_for_outbound_sockets=127.13.99.3
--webserver_interface=127.13.99.3
--webserver_port=0
--tserver_master_addrs=127.13.99.62:35893
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/krb5kdc/oryx_127.13.99.3.keytab
--principal=oryx/127.13.99.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:44777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:45:49.875919 26262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:45:49.876159 26262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:45:49.876201 26262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:45:49.880151 26262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:45:49.880208 26262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:45:49.880245 26262 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 09:45:49.880276 26262 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.99.3
I20260812 09:45:49.885447 26262 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:44777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/krb5kdc/oryx_127.13.99.3.keytab
--principal=oryx/127.13.99.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.99.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.99.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.99.62:35893
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26262
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.99.3
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:45:49.886477 26262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:45:49.887485 26262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:45:49.897428 26281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:49.897444 26284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:49.897440 26289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:45:49.897694 26262 server_base.cc:1034] running on GCE node
I20260812 09:45:49.898101 26262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:45:49.898746 26262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:45:49.899906 26262 hybrid_clock.cc:648] HybridClock initialized: now 1786527949899897 us; error 35 us; skew 500 ppm
Aug 12 09:45:49 dist-test-slave-jq4t krb5kdc[25661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527949, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.99.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:45:49.902689 26262 init.cc:377] Logged in from keytab as oryx/127.13.99.3@KRBTEST.COM (short username oryx)
I20260812 09:45:49.903544 26262 webserver.cc:460] Webserver started at http://127.13.99.3:42749/ using document root <none> and password file <none>
I20260812 09:45:49.904114 26262 fs_manager.cc:359] Metadata directory not provided
I20260812 09:45:49.904204 26262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:45:49.904446 26262 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:45:49.906702 26262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/minicluster-data/ts-2/data/instance:
uuid: "7a79b23b92bc49bda09c1b6e71137e49"
format_stamp: "Formatted at 2026-08-12 09:45:49 on dist-test-slave-jq4t"
server_key: "a7ff5b0ba8c143ddcde81b85a97fbcab"
server_key_iv: "4db62ee09d28502b0e9ed74ba8eb1a3f"
server_key_version: "encryptionkey@0"
I20260812 09:45:49.907269 26262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/minicluster-data/ts-2/wal/instance:
uuid: "7a79b23b92bc49bda09c1b6e71137e49"
format_stamp: "Formatted at 2026-08-12 09:45:49 on dist-test-slave-jq4t"
server_key: "a7ff5b0ba8c143ddcde81b85a97fbcab"
server_key_iv: "4db62ee09d28502b0e9ed74ba8eb1a3f"
server_key_version: "encryptionkey@0"
I20260812 09:45:49.910974 26262 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260812 09:45:49.913374 26303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:49.914583 26262 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 09:45:49.914713 26262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/minicluster-data/ts-2/wal
uuid: "7a79b23b92bc49bda09c1b6e71137e49"
format_stamp: "Formatted at 2026-08-12 09:45:49 on dist-test-slave-jq4t"
server_key: "a7ff5b0ba8c143ddcde81b85a97fbcab"
server_key_iv: "4db62ee09d28502b0e9ed74ba8eb1a3f"
server_key_version: "encryptionkey@0"
I20260812 09:45:49.914820 26262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:45:49.940415 26262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:45:49.943321 26262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:45:49.943533 26262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:45:49.945107 26262 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:45:49.945178 26262 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:49.945252 26262 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:45:49.945293 26262 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 09:45:49 dist-test-slave-jq4t krb5kdc[25661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527949, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.99.3@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM
I20260812 09:45:49.956265 26262 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.3:46139
I20260812 09:45:49.956348 26496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.3:46139 every 8 connection(s)
I20260812 09:45:49.957371 26262 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/minicluster-data/ts-2/data/info.pb
I20260812 09:45:49.959838 25803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:49.945892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.3:60687 (local address 127.13.99.62:35893)
0812 09:45:49.946048 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:49.946053 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:49.947031 (+ 978us) server_negotiation.cc:407] Connection header received
0812 09:45:49.948089 (+ 1058us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:49.948096 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:49.948162 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:49.948228 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:49.949810 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:49.950709 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:49.952382 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:49.952699 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:49.952836 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:49.955960 (+ 3124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:49.955964 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:49.955967 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:49.956000 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:49.957653 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:49.958231 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:49.958234 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:49.958236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:49.958327 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:49.958687 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:49.958691 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:49.958692 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:49.958892 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:49.959032 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:49.959541 (+ 509us) server_negotiation.cc:299] Negotiation successful
0812 09:45:49.959666 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260812 09:45:49.960642 26320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:49.946377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:35893 (local address 127.13.99.3:60687)
0812 09:45:49.946845 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:49.946895 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:49.947827 (+ 932us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:49.948720 (+ 893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:49.948727 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:49.949046 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:45:49.949626 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:49.949647 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:49.950874 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:49.950880 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:49.952246 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:49.952258 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:49.952898 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:49.952903 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:49.953085 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:49.954243 (+ 1158us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:45:49.954259 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:45:49.955784 (+ 1525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:45:49.957772 (+ 1988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:49.957778 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:49.957792 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:49.958127 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:49.958435 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:49.958438 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:49.958440 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:49.958583 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:49.959030 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:45:49.959037 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:45:49.959296 (+ 259us) client_negotiation.cc:769] Sending connection context
0812 09:45:49.959494 (+ 198us) client_negotiation.cc:240] Negotiation successful
0812 09:45:49.959800 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":133,"threads_started":1}
I20260812 09:45:49.961941 26498 heartbeater.cc:347] Connected to a master server at 127.13.99.62:35893
I20260812 09:45:49.962105 26498 heartbeater.cc:464] Registering TS with master...
I20260812 09:45:49.962620 26498 heartbeater.cc:511] Master 127.13.99.62:35893 requested a full tablet report, sending...
I20260812 09:45:49.963763 25720 ts_manager.cc:194] Registered new tserver with Master: 7a79b23b92bc49bda09c1b6e71137e49 (127.13.99.3:46139)
I20260812 09:45:49.964586 25720 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.99.3@KRBTEST.COM'} at 127.13.99.3:60687
I20260812 09:45:49.966859 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 26262
I20260812 09:45:49.967029 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/minicluster-data/ts-2/wal/instance
I20260812 09:45:49.967407 13708 external_mini_cluster.cc:1428] Setting key 8dd5712182eb69f7e7c231af83559681
I20260812 09:45:49.969637 13708 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 09:45:49.971377 25803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:49.964151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.3:47945 (local address 127.13.99.62:35893)
0812 09:45:49.964270 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:49.964274 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:49.964413 (+ 139us) server_negotiation.cc:407] Connection header received
0812 09:45:49.964547 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:49.964550 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:49.964586 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:49.964650 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:49.965141 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:49.965675 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:49.966620 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:49.966894 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:49.967017 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:49.968167 (+ 1150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:49.968170 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:49.968173 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:49.968197 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:49.969747 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:49.970172 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:49.970175 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:49.970177 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:49.970260 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:49.970554 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:49.970557 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:49.970559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:49.970733 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:49.970855 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:49.971116 (+ 261us) server_negotiation.cc:299] Negotiation successful
0812 09:45:49.971221 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260812 09:45:49.971411 26508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:49.964023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:35893 (local address 127.13.99.3:47945)
0812 09:45:49.964326 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:49.964341 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:49.964433 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:49.964702 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:49.964705 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:49.964934 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:45:49.965032 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:49.965038 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:49.965833 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:49.965836 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:49.966477 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:49.966486 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:49.967065 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:49.967069 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:49.967170 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:49.967701 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:45:49.967719 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:45:49.968026 (+ 307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:45:49.969865 (+ 1839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:49.969868 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:49.969870 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:49.970069 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:49.970356 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:49.970359 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:49.970361 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:49.970456 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:49.970846 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:45:49.970849 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:45:49.970948 (+ 99us) client_negotiation.cc:769] Sending connection context
0812 09:45:49.971058 (+ 110us) client_negotiation.cc:240] Negotiation successful
0812 09:45:49.971203 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":102,"threads_started":1}
Aug 12 09:45:49 dist-test-slave-jq4t krb5kdc[25661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527949, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 09:45:49 dist-test-slave-jq4t krb5kdc[25661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 09:45:49 dist-test-slave-jq4t krb5kdc[25661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
W20260812 09:45:49.996052 25803 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 09:45:49.989720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55114 (local address 127.13.99.62:35893)
0812 09:45:49.989856 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:49.989859 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:49.989871 (+ 12us) server_negotiation.cc:407] Connection header received
0812 09:45:49.989911 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:49.989914 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:49.989954 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:49.990018 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:49.990737 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:49.991296 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:49.992189 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:49.992401 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:49.992589 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:49.995856 (+ 3267us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55114: BlockingRecv error: recv got EOF from 127.0.0.1:55114 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 12 09:45:49 dist-test-slave-jq4t krb5kdc[25661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527949, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM
I20260812 09:45:50.005939 25803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:49.997588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55124 (local address 127.13.99.62:35893)
0812 09:45:49.997717 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:49.997727 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:49.997878 (+ 151us) server_negotiation.cc:407] Connection header received
0812 09:45:49.998003 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:49.998005 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:49.998048 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:49.998172 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:49.998745 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:49.999266 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:50.000044 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:50.000273 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:50.000366 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:50.002700 (+ 2334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:50.002703 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:50.002705 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:50.002730 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:50.004039 (+ 1309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:50.004697 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:50.004700 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:50.004701 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:50.004788 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:50.005121 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:50.005124 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:50.005125 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:50.005316 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:50.005439 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:50.005709 (+ 270us) server_negotiation.cc:299] Negotiation successful
0812 09:45:50.005817 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260812 09:45:50.009505 25720 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55124:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 09:45:50.012672 25720 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 09:45:50.027446 26536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:50.021414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.1:35609 (local address 127.0.0.1:48072)
0812 09:45:50.021961 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:50.022043 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:50.022167 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:50.022963 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:50.022970 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:50.022991 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:45:50.023133 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:50.023141 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:50.024420 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:50.024425 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:50.025741 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:50.025752 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:50.026876 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:50.026879 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:50.026984 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:50.027028 (+ 44us) client_negotiation.cc:769] Sending connection context
0812 09:45:50.027149 (+ 121us) client_negotiation.cc:240] Negotiation successful
0812 09:45:50.027292 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"mutex_wait_us":23,"thread_start_us":98,"threads_started":1}
I20260812 09:45:50.027454 26539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:50.021837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48072 (local address 127.13.99.1:35609)
0812 09:45:50.022138 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:50.022147 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:50.022177 (+ 30us) server_negotiation.cc:407] Connection header received
0812 09:45:50.022366 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:50.022374 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:50.022606 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:50.022770 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:45:50.023276 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:50.024308 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:50.025879 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:50.026741 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:50.026856 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:50.027004 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:50.027197 (+ 193us) server_negotiation.cc:299] Negotiation successful
0812 09:45:50.027302 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":122,"threads_started":1}
I20260812 09:45:50.027690 26537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:50.022151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.2:36055 (local address 127.0.0.1:54622)
0812 09:45:50.022791 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:50.022803 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:50.022887 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:50.023559 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:50.023562 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:50.023584 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:45:50.023725 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:50.023732 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:50.025180 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:50.025184 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:50.026311 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:50.026320 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:50.027303 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:50.027306 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:50.027401 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:50.027424 (+ 23us) client_negotiation.cc:769] Sending connection context
0812 09:45:50.027475 (+ 51us) client_negotiation.cc:240] Negotiation successful
0812 09:45:50.027548 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"mutex_wait_us":25,"thread_start_us":103,"threads_started":1}
I20260812 09:45:50.027951 26541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:50.021596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.3:46139 (local address 127.0.0.1:41666)
0812 09:45:50.022505 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:50.022519 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:50.022607 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:50.023634 (+ 1027us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:50.023637 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:50.023665 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:45:50.023833 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:50.023842 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:50.025317 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:50.025324 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:50.026457 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:50.026467 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:50.027527 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:50.027532 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:50.027645 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:50.027670 (+ 25us) client_negotiation.cc:769] Sending connection context
0812 09:45:50.027724 (+ 54us) client_negotiation.cc:240] Negotiation successful
0812 09:45:50.027798 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":844,"mutex_wait_us":51,"spinlock_wait_cycles":896,"thread_start_us":102,"threads_started":1}
I20260812 09:45:50.028275 26546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:50.022569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54622 (local address 127.13.99.2:36055)
0812 09:45:50.023138 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:50.023150 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:50.023173 (+ 23us) server_negotiation.cc:407] Connection header received
0812 09:45:50.023248 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:50.023253 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:50.023430 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:50.023589 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:45:50.023858 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:50.025059 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:50.026443 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:50.027172 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:50.027818 (+ 646us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:50.027998 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:50.028052 (+ 54us) server_negotiation.cc:299] Negotiation successful
0812 09:45:50.028144 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":112,"threads_started":1}
I20260812 09:45:50.028759 26543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:50.022253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41666 (local address 127.13.99.3:46139)
0812 09:45:50.023137 (+ 884us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:50.023150 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:50.023173 (+ 23us) server_negotiation.cc:407] Connection header received
0812 09:45:50.023248 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:50.023253 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:50.023430 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:50.023984 (+ 554us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:45:50.024121 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:50.025211 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:50.026589 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:50.027370 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:50.028225 (+ 855us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:50.028410 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:50.028465 (+ 55us) server_negotiation.cc:299] Negotiation successful
0812 09:45:50.028621 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":765,"thread_start_us":123,"threads_started":1}
I20260812 09:45:50.030933 26140 tablet_service.cc:1467] Processing CreateTablet for tablet 45097a1ff9ff4e4485104ad50a03f8f2 (DEFAULT_TABLE table=test-table [id=db6c89950238450788907c13177cd49b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 09:45:50.031059 26394 tablet_service.cc:1467] Processing CreateTablet for tablet 45097a1ff9ff4e4485104ad50a03f8f2 (DEFAULT_TABLE table=test-table [id=db6c89950238450788907c13177cd49b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 09:45:50.031059 25936 tablet_service.cc:1467] Processing CreateTablet for tablet 45097a1ff9ff4e4485104ad50a03f8f2 (DEFAULT_TABLE table=test-table [id=db6c89950238450788907c13177cd49b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 09:45:50.031960 26140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45097a1ff9ff4e4485104ad50a03f8f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:45:50.032074 26394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45097a1ff9ff4e4485104ad50a03f8f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:45:50.032071 25936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45097a1ff9ff4e4485104ad50a03f8f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:45:50.039106 26550 tablet_bootstrap.cc:492] T 45097a1ff9ff4e4485104ad50a03f8f2 P 49fa8338699749498ff4283df5548ab3: Bootstrap starting.
I20260812 09:45:50.040131 26552 tablet_bootstrap.cc:492] T 45097a1ff9ff4e4485104ad50a03f8f2 P 7a79b23b92bc49bda09c1b6e71137e49: Bootstrap starting.
I20260812 09:45:50.040244 26551 tablet_bootstrap.cc:492] T 45097a1ff9ff4e4485104ad50a03f8f2 P c0742b25a62b4793bf20353f0d1f2a95: Bootstrap starting.
I20260812 09:45:50.041929 26550 tablet_bootstrap.cc:654] T 45097a1ff9ff4e4485104ad50a03f8f2 P 49fa8338699749498ff4283df5548ab3: Neither blocks nor log segments found. Creating new log.
I20260812 09:45:50.042418 26551 tablet_bootstrap.cc:654] T 45097a1ff9ff4e4485104ad50a03f8f2 P c0742b25a62b4793bf20353f0d1f2a95: Neither blocks nor log segments found. Creating new log.
I20260812 09:45:50.042418 26552 tablet_bootstrap.cc:654] T 45097a1ff9ff4e4485104ad50a03f8f2 P 7a79b23b92bc49bda09c1b6e71137e49: Neither blocks nor log segments found. Creating new log.
I20260812 09:45:50.042887 26550 log.cc:824] T 45097a1ff9ff4e4485104ad50a03f8f2 P 49fa8338699749498ff4283df5548ab3: Log is configured to *not* fsync() on all Append() calls
I20260812 09:45:50.043293 26551 log.cc:824] T 45097a1ff9ff4e4485104ad50a03f8f2 P c0742b25a62b4793bf20353f0d1f2a95: Log is configured to *not* fsync() on all Append() calls
I20260812 09:45:50.043293 26552 log.cc:824] T 45097a1ff9ff4e4485104ad50a03f8f2 P 7a79b23b92bc49bda09c1b6e71137e49: Log is configured to *not* fsync() on all Append() calls
I20260812 09:45:50.045518 26551 tablet_bootstrap.cc:492] T 45097a1ff9ff4e4485104ad50a03f8f2 P c0742b25a62b4793bf20353f0d1f2a95: No bootstrap required, opened a new log
I20260812 09:45:50.045528 26552 tablet_bootstrap.cc:492] T 45097a1ff9ff4e4485104ad50a03f8f2 P 7a79b23b92bc49bda09c1b6e71137e49: No bootstrap required, opened a new log
I20260812 09:45:50.045550 26550 tablet_bootstrap.cc:492] T 45097a1ff9ff4e4485104ad50a03f8f2 P 49fa8338699749498ff4283df5548ab3: No bootstrap required, opened a new log
I20260812 09:45:50.045733 26552 ts_tablet_manager.cc:1397] T 45097a1ff9ff4e4485104ad50a03f8f2 P 7a79b23b92bc49bda09c1b6e71137e49: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260812 09:45:50.045733 26551 ts_tablet_manager.cc:1397] T 45097a1ff9ff4e4485104ad50a03f8f2 P c0742b25a62b4793bf20353f0d1f2a95: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260812 09:45:50.045769 26550 ts_tablet_manager.cc:1397] T 45097a1ff9ff4e4485104ad50a03f8f2 P 49fa8338699749498ff4283df5548ab3: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260812 09:45:50.049167 26550 raft_consensus.cc:348] T 45097a1ff9ff4e4485104ad50a03f8f2 P 49fa8338699749498ff4283df5548ab3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0742b25a62b4793bf20353f0d1f2a95" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 35609 } } peers { permanent_uuid: "7a79b23b92bc49bda09c1b6e71137e49" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 46139 } } peers { permanent_uuid: "49fa8338699749498ff4283df5548ab3" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 36055 } }
I20260812 09:45:50.049443 26550 raft_consensus.cc:374] T 45097a1ff9ff4e4485104ad50a03f8f2 P 49fa8338699749498ff4283df5548ab3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:45:50.049523 26550 raft_consensus.cc:729] T 45097a1ff9ff4e4485104ad50a03f8f2 P 49fa8338699749498ff4283df5548ab3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49fa8338699749498ff4283df5548ab3, State: Initialized, Role: FOLLOWER
I20260812 09:45:50.050061 26550 consensus_queue.cc:260] T 45097a1ff9ff4e4485104ad50a03f8f2 P 49fa8338699749498ff4283df5548ab3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0742b25a62b4793bf20353f0d1f2a95" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 35609 } } peers { permanent_uuid: "7a79b23b92bc49bda09c1b6e71137e49" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 46139 } } peers { permanent_uuid: "49fa8338699749498ff4283df5548ab3" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 36055 } }
I20260812 09:45:50.050366 26552 raft_consensus.cc:348] T 45097a1ff9ff4e4485104ad50a03f8f2 P 7a79b23b92bc49bda09c1b6e71137e49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0742b25a62b4793bf20353f0d1f2a95" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 35609 } } peers { permanent_uuid: "7a79b23b92bc49bda09c1b6e71137e49" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 46139 } } peers { permanent_uuid: "49fa8338699749498ff4283df5548ab3" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 36055 } }
I20260812 09:45:50.050683 26552 raft_consensus.cc:374] T 45097a1ff9ff4e4485104ad50a03f8f2 P 7a79b23b92bc49bda09c1b6e71137e49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:45:50.050783 26552 raft_consensus.cc:729] T 45097a1ff9ff4e4485104ad50a03f8f2 P 7a79b23b92bc49bda09c1b6e71137e49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a79b23b92bc49bda09c1b6e71137e49, State: Initialized, Role: FOLLOWER
I20260812 09:45:50.051353 26550 ts_tablet_manager.cc:1428] T 45097a1ff9ff4e4485104ad50a03f8f2 P 49fa8338699749498ff4283df5548ab3: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260812 09:45:50.051272 26552 consensus_queue.cc:260] T 45097a1ff9ff4e4485104ad50a03f8f2 P 7a79b23b92bc49bda09c1b6e71137e49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0742b25a62b4793bf20353f0d1f2a95" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 35609 } } peers { permanent_uuid: "7a79b23b92bc49bda09c1b6e71137e49" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 46139 } } peers { permanent_uuid: "49fa8338699749498ff4283df5548ab3" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 36055 } }
I20260812 09:45:50.051187 26551 raft_consensus.cc:348] T 45097a1ff9ff4e4485104ad50a03f8f2 P c0742b25a62b4793bf20353f0d1f2a95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0742b25a62b4793bf20353f0d1f2a95" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 35609 } } peers { permanent_uuid: "7a79b23b92bc49bda09c1b6e71137e49" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 46139 } } peers { permanent_uuid: "49fa8338699749498ff4283df5548ab3" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 36055 } }
I20260812 09:45:50.051443 26551 raft_consensus.cc:374] T 45097a1ff9ff4e4485104ad50a03f8f2 P c0742b25a62b4793bf20353f0d1f2a95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:45:50.051548 26551 raft_consensus.cc:729] T 45097a1ff9ff4e4485104ad50a03f8f2 P c0742b25a62b4793bf20353f0d1f2a95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0742b25a62b4793bf20353f0d1f2a95, State: Initialized, Role: FOLLOWER
I20260812 09:45:50.051764 26247 heartbeater.cc:503] Master 127.13.99.62:35893 was elected leader, sending a full tablet report...
I20260812 09:45:50.052034 26551 consensus_queue.cc:260] T 45097a1ff9ff4e4485104ad50a03f8f2 P c0742b25a62b4793bf20353f0d1f2a95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0742b25a62b4793bf20353f0d1f2a95" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 35609 } } peers { permanent_uuid: "7a79b23b92bc49bda09c1b6e71137e49" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 46139 } } peers { permanent_uuid: "49fa8338699749498ff4283df5548ab3" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 36055 } }
I20260812 09:45:50.052188 26498 heartbeater.cc:503] Master 127.13.99.62:35893 was elected leader, sending a full tablet report...
I20260812 09:45:50.052687 26032 heartbeater.cc:503] Master 127.13.99.62:35893 was elected leader, sending a full tablet report...
I20260812 09:45:50.052965 26551 ts_tablet_manager.cc:1428] T 45097a1ff9ff4e4485104ad50a03f8f2 P c0742b25a62b4793bf20353f0d1f2a95: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260812 09:45:50.053462 26552 ts_tablet_manager.cc:1428] T 45097a1ff9ff4e4485104ad50a03f8f2 P 7a79b23b92bc49bda09c1b6e71137e49: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260812 09:45:50.111619 26564 raft_consensus.cc:482] T 45097a1ff9ff4e4485104ad50a03f8f2 P c0742b25a62b4793bf20353f0d1f2a95 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 09:45:50.111892 26564 raft_consensus.cc:504] T 45097a1ff9ff4e4485104ad50a03f8f2 P c0742b25a62b4793bf20353f0d1f2a95 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0742b25a62b4793bf20353f0d1f2a95" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 35609 } } peers { permanent_uuid: "7a79b23b92bc49bda09c1b6e71137e49" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 46139 } } peers { permanent_uuid: "49fa8338699749498ff4283df5548ab3" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 36055 } }
I20260812 09:45:50.113092 26564 leader_election.cc:288] T 45097a1ff9ff4e4485104ad50a03f8f2 P c0742b25a62b4793bf20353f0d1f2a95 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a79b23b92bc49bda09c1b6e71137e49 (127.13.99.3:46139), 49fa8338699749498ff4283df5548ab3 (127.13.99.2:36055)
I20260812 09:45:50.118191 26546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:50.114008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.1:36803 (local address 127.13.99.2:36055)
0812 09:45:50.114139 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:50.114144 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:50.114244 (+ 100us) server_negotiation.cc:407] Connection header received
0812 09:45:50.114368 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:50.114371 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:50.114424 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:50.114507 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:45:50.114769 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:50.115485 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:50.117027 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:50.117621 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:50.117733 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:50.117799 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:50.117991 (+ 192us) server_negotiation.cc:299] Negotiation successful
0812 09:45:50.118058 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260812 09:45:50.118232 26543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:50.113505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.1:37725 (local address 127.13.99.3:46139)
0812 09:45:50.113670 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:50.113676 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:50.113824 (+ 148us) server_negotiation.cc:407] Connection header received
0812 09:45:50.114034 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:50.114038 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:50.114111 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:50.114195 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:45:50.114541 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:50.115344 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:50.117075 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:50.117636 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:50.117751 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:50.117818 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:50.118031 (+ 213us) server_negotiation.cc:299] Negotiation successful
0812 09:45:50.118105 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260812 09:45:50.118259 26566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:50.113352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.3:46139 (local address 127.13.99.1:37725)
0812 09:45:50.113725 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:50.113738 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:50.113878 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:50.114217 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:50.114221 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:50.114263 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:45:50.114415 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:50.114421 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:50.115456 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:50.115458 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:50.116896 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:50.116906 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:50.117767 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:50.117771 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:50.117897 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:50.117916 (+ 19us) client_negotiation.cc:769] Sending connection context
0812 09:45:50.118002 (+ 86us) client_negotiation.cc:240] Negotiation successful
0812 09:45:50.118080 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"mutex_wait_us":47,"spinlock_wait_cycles":7808,"thread_start_us":144,"threads_started":1}
I20260812 09:45:50.118284 26567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:50.113841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.2:36055 (local address 127.13.99.1:36803)
0812 09:45:50.114137 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:50.114151 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:50.114263 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:50.114530 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:50.114535 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:50.114557 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:45:50.114659 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:50.114664 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:50.115582 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:50.115586 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:50.116894 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:50.116906 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:50.117735 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:50.117739 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:50.117848 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:50.117871 (+ 23us) client_negotiation.cc:769] Sending connection context
0812 09:45:50.117974 (+ 103us) client_negotiation.cc:240] Negotiation successful
0812 09:45:50.118049 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"mutex_wait_us":35,"spinlock_wait_cycles":32512,"thread_start_us":144,"threads_started":1}
I20260812 09:45:50.119158 26180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45097a1ff9ff4e4485104ad50a03f8f2" candidate_uuid: "c0742b25a62b4793bf20353f0d1f2a95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49fa8338699749498ff4283df5548ab3" is_pre_election: true
I20260812 09:45:50.119206 26424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45097a1ff9ff4e4485104ad50a03f8f2" candidate_uuid: "c0742b25a62b4793bf20353f0d1f2a95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a79b23b92bc49bda09c1b6e71137e49" is_pre_election: true
I20260812 09:45:50.119491 26424 raft_consensus.cc:2454] T 45097a1ff9ff4e4485104ad50a03f8f2 P 7a79b23b92bc49bda09c1b6e71137e49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0742b25a62b4793bf20353f0d1f2a95 in term 0.
I20260812 09:45:50.119508 26180 raft_consensus.cc:2454] T 45097a1ff9ff4e4485104ad50a03f8f2 P 49fa8338699749498ff4283df5548ab3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0742b25a62b4793bf20353f0d1f2a95 in term 0.
I20260812 09:45:50.119980 25863 leader_election.cc:302] T 45097a1ff9ff4e4485104ad50a03f8f2 P c0742b25a62b4793bf20353f0d1f2a95 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49fa8338699749498ff4283df5548ab3, c0742b25a62b4793bf20353f0d1f2a95; no voters:
I20260812 09:45:50.120272 26564 raft_consensus.cc:2781] T 45097a1ff9ff4e4485104ad50a03f8f2 P c0742b25a62b4793bf20353f0d1f2a95 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 09:45:50.120349 26564 raft_consensus.cc:482] T 45097a1ff9ff4e4485104ad50a03f8f2 P c0742b25a62b4793bf20353f0d1f2a95 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 09:45:50.120383 26564 raft_consensus.cc:3037] T 45097a1ff9ff4e4485104ad50a03f8f2 P c0742b25a62b4793bf20353f0d1f2a95 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:45:50.121510 26564 raft_consensus.cc:504] T 45097a1ff9ff4e4485104ad50a03f8f2 P c0742b25a62b4793bf20353f0d1f2a95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0742b25a62b4793bf20353f0d1f2a95" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 35609 } } peers { permanent_uuid: "7a79b23b92bc49bda09c1b6e71137e49" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 46139 } } peers { permanent_uuid: "49fa8338699749498ff4283df5548ab3" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 36055 } }
I20260812 09:45:50.121841 26564 leader_election.cc:288] T 45097a1ff9ff4e4485104ad50a03f8f2 P c0742b25a62b4793bf20353f0d1f2a95 [CANDIDATE]: Term 1 election: Requested vote from peers 7a79b23b92bc49bda09c1b6e71137e49 (127.13.99.3:46139), 49fa8338699749498ff4283df5548ab3 (127.13.99.2:36055)
I20260812 09:45:50.122262 26424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45097a1ff9ff4e4485104ad50a03f8f2" candidate_uuid: "c0742b25a62b4793bf20353f0d1f2a95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a79b23b92bc49bda09c1b6e71137e49"
I20260812 09:45:50.122382 26180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45097a1ff9ff4e4485104ad50a03f8f2" candidate_uuid: "c0742b25a62b4793bf20353f0d1f2a95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49fa8338699749498ff4283df5548ab3"
I20260812 09:45:50.122390 26424 raft_consensus.cc:3037] T 45097a1ff9ff4e4485104ad50a03f8f2 P 7a79b23b92bc49bda09c1b6e71137e49 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:45:50.122541 26180 raft_consensus.cc:3037] T 45097a1ff9ff4e4485104ad50a03f8f2 P 49fa8338699749498ff4283df5548ab3 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:45:50.123711 26424 raft_consensus.cc:2454] T 45097a1ff9ff4e4485104ad50a03f8f2 P 7a79b23b92bc49bda09c1b6e71137e49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0742b25a62b4793bf20353f0d1f2a95 in term 1.
I20260812 09:45:50.124042 25863 leader_election.cc:302] T 45097a1ff9ff4e4485104ad50a03f8f2 P c0742b25a62b4793bf20353f0d1f2a95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a79b23b92bc49bda09c1b6e71137e49, c0742b25a62b4793bf20353f0d1f2a95; no voters:
I20260812 09:45:50.124250 26564 raft_consensus.cc:2781] T 45097a1ff9ff4e4485104ad50a03f8f2 P c0742b25a62b4793bf20353f0d1f2a95 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 09:45:50.124281 26180 raft_consensus.cc:2454] T 45097a1ff9ff4e4485104ad50a03f8f2 P 49fa8338699749498ff4283df5548ab3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0742b25a62b4793bf20353f0d1f2a95 in term 1.
I20260812 09:45:50.124539 26564 raft_consensus.cc:686] T 45097a1ff9ff4e4485104ad50a03f8f2 P c0742b25a62b4793bf20353f0d1f2a95 [term 1 LEADER]: Becoming Leader. State: Replica: c0742b25a62b4793bf20353f0d1f2a95, State: Running, Role: LEADER
I20260812 09:45:50.124894 26564 consensus_queue.cc:237] T 45097a1ff9ff4e4485104ad50a03f8f2 P c0742b25a62b4793bf20353f0d1f2a95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0742b25a62b4793bf20353f0d1f2a95" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 35609 } } peers { permanent_uuid: "7a79b23b92bc49bda09c1b6e71137e49" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 46139 } } peers { permanent_uuid: "49fa8338699749498ff4283df5548ab3" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 36055 } }
I20260812 09:45:50.128341 25720 catalog_manager.cc:5495] T 45097a1ff9ff4e4485104ad50a03f8f2 P c0742b25a62b4793bf20353f0d1f2a95 reported cstate change: term changed from 0 to 1, leader changed from <none> to c0742b25a62b4793bf20353f0d1f2a95 (127.13.99.1). New cstate: current_term: 1 leader_uuid: "c0742b25a62b4793bf20353f0d1f2a95" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0742b25a62b4793bf20353f0d1f2a95" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 35609 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a79b23b92bc49bda09c1b6e71137e49" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 46139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49fa8338699749498ff4283df5548ab3" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 36055 } health_report { overall_health: UNKNOWN } } }
I20260812 09:45:50.149561 26539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:50.146275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48074 (local address 127.13.99.1:35609)
0812 09:45:50.146412 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:50.146415 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:50.146429 (+ 14us) server_negotiation.cc:407] Connection header received
0812 09:45:50.146526 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:50.146529 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:50.146584 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:50.146672 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 09:45:50.146887 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:50.147392 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:50.148189 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:50.148393 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:50.148467 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:50.148752 (+ 285us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 09:45:50.149186 (+ 434us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 09:45:50.149275 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:50.149331 (+ 56us) server_negotiation.cc:299] Negotiation successful
0812 09:45:50.149380 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260812 09:45:50.155427 26180 raft_consensus.cc:1261] T 45097a1ff9ff4e4485104ad50a03f8f2 P 49fa8338699749498ff4283df5548ab3 [term 1 FOLLOWER]: Refusing update from remote peer c0742b25a62b4793bf20353f0d1f2a95: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 09:45:50.155689 26424 raft_consensus.cc:1261] T 45097a1ff9ff4e4485104ad50a03f8f2 P 7a79b23b92bc49bda09c1b6e71137e49 [term 1 FOLLOWER]: Refusing update from remote peer c0742b25a62b4793bf20353f0d1f2a95: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 09:45:50.156186 26568 consensus_queue.cc:1035] T 45097a1ff9ff4e4485104ad50a03f8f2 P c0742b25a62b4793bf20353f0d1f2a95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49fa8338699749498ff4283df5548ab3" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 36055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 09:45:50.156361 26564 consensus_queue.cc:1035] T 45097a1ff9ff4e4485104ad50a03f8f2 P c0742b25a62b4793bf20353f0d1f2a95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a79b23b92bc49bda09c1b6e71137e49" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 46139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 09:45:50.172344 25724 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55124:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 09:45:50.172561 25724 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55124:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 09:45:50.175065 25724 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7fb7f7ee6bd04ed7bb41ec750ae6d72a: Sending DeleteTablet for 3 replicas of tablet 45097a1ff9ff4e4485104ad50a03f8f2
I20260812 09:45:50.175992 26394 tablet_service.cc:1514] Processing DeleteTablet for tablet 45097a1ff9ff4e4485104ad50a03f8f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 09:45:50 UTC) from {username='oryx', principal='oryx/127.13.99.62@KRBTEST.COM'} at 127.0.0.1:41666
I20260812 09:45:50.176014 25936 tablet_service.cc:1514] Processing DeleteTablet for tablet 45097a1ff9ff4e4485104ad50a03f8f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 09:45:50 UTC) from {username='oryx', principal='oryx/127.13.99.62@KRBTEST.COM'} at 127.0.0.1:48072
I20260812 09:45:50.176005 26140 tablet_service.cc:1514] Processing DeleteTablet for tablet 45097a1ff9ff4e4485104ad50a03f8f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 09:45:50 UTC) from {username='oryx', principal='oryx/127.13.99.62@KRBTEST.COM'} at 127.0.0.1:54622
I20260812 09:45:50.176510 26594 tablet_replica.cc:330] T 45097a1ff9ff4e4485104ad50a03f8f2 P 49fa8338699749498ff4283df5548ab3: stopping tablet replica
I20260812 09:45:50.176833 26594 raft_consensus.cc:2229] T 45097a1ff9ff4e4485104ad50a03f8f2 P 49fa8338699749498ff4283df5548ab3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 09:45:50.177150 26594 raft_consensus.cc:2258] T 45097a1ff9ff4e4485104ad50a03f8f2 P 49fa8338699749498ff4283df5548ab3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 09:45:50.177258 25724 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55124:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 09:45:50.177700 25724 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 09:45:50.178668 26594 ts_tablet_manager.cc:1905] T 45097a1ff9ff4e4485104ad50a03f8f2 P 49fa8338699749498ff4283df5548ab3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 09:45:50.181571 26594 ts_tablet_manager.cc:1918] T 45097a1ff9ff4e4485104ad50a03f8f2 P 49fa8338699749498ff4283df5548ab3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 09:45:50.181675 26594 log.cc:1196] T 45097a1ff9ff4e4485104ad50a03f8f2 P 49fa8338699749498ff4283df5548ab3: Deleting WAL directory at /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/minicluster-data/ts-1/wal/wals/45097a1ff9ff4e4485104ad50a03f8f2
I20260812 09:45:50.182029 26594 ts_tablet_manager.cc:1939] T 45097a1ff9ff4e4485104ad50a03f8f2 P 49fa8338699749498ff4283df5548ab3: Deleting consensus metadata
I20260812 09:45:50.182446 25936 tablet_service.cc:1467] Processing CreateTablet for tablet 69856a9323ad48e4bd81faca5ce84fec (DEFAULT_TABLE table=test-table [id=2de3e16de8df4884b29bcc5f7fcca730]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 09:45:50.182538 26140 tablet_service.cc:1467] Processing CreateTablet for tablet 69856a9323ad48e4bd81faca5ce84fec (DEFAULT_TABLE table=test-table [id=2de3e16de8df4884b29bcc5f7fcca730]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 09:45:50.182741 25936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69856a9323ad48e4bd81faca5ce84fec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:45:50.182765 26140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69856a9323ad48e4bd81faca5ce84fec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:45:50.183283 25701 catalog_manager.cc:4841] TS 49fa8338699749498ff4283df5548ab3 (127.13.99.2:36055): tablet 45097a1ff9ff4e4485104ad50a03f8f2 (table test-table [id=db6c89950238450788907c13177cd49b]) successfully deleted
I20260812 09:45:50.184680 26593 tablet_replica.cc:330] T 45097a1ff9ff4e4485104ad50a03f8f2 P 7a79b23b92bc49bda09c1b6e71137e49: stopping tablet replica
I20260812 09:45:50.184676 26592 tablet_replica.cc:330] T 45097a1ff9ff4e4485104ad50a03f8f2 P c0742b25a62b4793bf20353f0d1f2a95: stopping tablet replica
I20260812 09:45:50.184965 26593 raft_consensus.cc:2229] T 45097a1ff9ff4e4485104ad50a03f8f2 P 7a79b23b92bc49bda09c1b6e71137e49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 09:45:50.184965 26592 raft_consensus.cc:2229] T 45097a1ff9ff4e4485104ad50a03f8f2 P c0742b25a62b4793bf20353f0d1f2a95 [term 1 LEADER]: Raft consensus shutting down.
I20260812 09:45:50.185060 26550 tablet_bootstrap.cc:492] T 69856a9323ad48e4bd81faca5ce84fec P 49fa8338699749498ff4283df5548ab3: Bootstrap starting.
I20260812 09:45:50.185173 26551 tablet_bootstrap.cc:492] T 69856a9323ad48e4bd81faca5ce84fec P c0742b25a62b4793bf20353f0d1f2a95: Bootstrap starting.
I20260812 09:45:50.185294 26592 raft_consensus.cc:2258] T 45097a1ff9ff4e4485104ad50a03f8f2 P c0742b25a62b4793bf20353f0d1f2a95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 09:45:50.185789 26394 tablet_service.cc:1467] Processing CreateTablet for tablet 69856a9323ad48e4bd81faca5ce84fec (DEFAULT_TABLE table=test-table [id=2de3e16de8df4884b29bcc5f7fcca730]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 09:45:50.186058 26550 tablet_bootstrap.cc:654] T 69856a9323ad48e4bd81faca5ce84fec P 49fa8338699749498ff4283df5548ab3: Neither blocks nor log segments found. Creating new log.
I20260812 09:45:50.186069 26551 tablet_bootstrap.cc:654] T 69856a9323ad48e4bd81faca5ce84fec P c0742b25a62b4793bf20353f0d1f2a95: Neither blocks nor log segments found. Creating new log.
I20260812 09:45:50.186074 26394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69856a9323ad48e4bd81faca5ce84fec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:45:50.186905 26593 raft_consensus.cc:2258] T 45097a1ff9ff4e4485104ad50a03f8f2 P 7a79b23b92bc49bda09c1b6e71137e49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 09:45:50.187325 26550 tablet_bootstrap.cc:492] T 69856a9323ad48e4bd81faca5ce84fec P 49fa8338699749498ff4283df5548ab3: No bootstrap required, opened a new log
I20260812 09:45:50.187414 26550 ts_tablet_manager.cc:1397] T 69856a9323ad48e4bd81faca5ce84fec P 49fa8338699749498ff4283df5548ab3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 09:45:50.187866 26550 raft_consensus.cc:348] T 69856a9323ad48e4bd81faca5ce84fec P 49fa8338699749498ff4283df5548ab3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0742b25a62b4793bf20353f0d1f2a95" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 35609 } } peers { permanent_uuid: "49fa8338699749498ff4283df5548ab3" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 36055 } } peers { permanent_uuid: "7a79b23b92bc49bda09c1b6e71137e49" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 46139 } }
I20260812 09:45:50.187975 26550 raft_consensus.cc:374] T 69856a9323ad48e4bd81faca5ce84fec P 49fa8338699749498ff4283df5548ab3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:45:50.188004 26550 raft_consensus.cc:729] T 69856a9323ad48e4bd81faca5ce84fec P 49fa8338699749498ff4283df5548ab3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49fa8338699749498ff4283df5548ab3, State: Initialized, Role: FOLLOWER
I20260812 09:45:50.188118 26550 consensus_queue.cc:260] T 69856a9323ad48e4bd81faca5ce84fec P 49fa8338699749498ff4283df5548ab3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0742b25a62b4793bf20353f0d1f2a95" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 35609 } } peers { permanent_uuid: "49fa8338699749498ff4283df5548ab3" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 36055 } } peers { permanent_uuid: "7a79b23b92bc49bda09c1b6e71137e49" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 46139 } }
I20260812 09:45:50.188375 26550 ts_tablet_manager.cc:1428] T 69856a9323ad48e4bd81faca5ce84fec P 49fa8338699749498ff4283df5548ab3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 09:45:50.188462 26593 ts_tablet_manager.cc:1905] T 45097a1ff9ff4e4485104ad50a03f8f2 P 7a79b23b92bc49bda09c1b6e71137e49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 09:45:50.190663 26552 tablet_bootstrap.cc:492] T 69856a9323ad48e4bd81faca5ce84fec P 7a79b23b92bc49bda09c1b6e71137e49: Bootstrap starting.
I20260812 09:45:50.190975 26593 ts_tablet_manager.cc:1918] T 45097a1ff9ff4e4485104ad50a03f8f2 P 7a79b23b92bc49bda09c1b6e71137e49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 09:45:50.191061 26593 log.cc:1196] T 45097a1ff9ff4e4485104ad50a03f8f2 P 7a79b23b92bc49bda09c1b6e71137e49: Deleting WAL directory at /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/minicluster-data/ts-2/wal/wals/45097a1ff9ff4e4485104ad50a03f8f2
I20260812 09:45:50.191361 26593 ts_tablet_manager.cc:1939] T 45097a1ff9ff4e4485104ad50a03f8f2 P 7a79b23b92bc49bda09c1b6e71137e49: Deleting consensus metadata
I20260812 09:45:50.191735 26552 tablet_bootstrap.cc:654] T 69856a9323ad48e4bd81faca5ce84fec P 7a79b23b92bc49bda09c1b6e71137e49: Neither blocks nor log segments found. Creating new log.
I20260812 09:45:50.192587 25701 catalog_manager.cc:4841] TS 7a79b23b92bc49bda09c1b6e71137e49 (127.13.99.3:46139): tablet 45097a1ff9ff4e4485104ad50a03f8f2 (table test-table [id=db6c89950238450788907c13177cd49b]) successfully deleted
I20260812 09:45:50.193312 26552 tablet_bootstrap.cc:492] T 69856a9323ad48e4bd81faca5ce84fec P 7a79b23b92bc49bda09c1b6e71137e49: No bootstrap required, opened a new log
I20260812 09:45:50.193397 26552 ts_tablet_manager.cc:1397] T 69856a9323ad48e4bd81faca5ce84fec P 7a79b23b92bc49bda09c1b6e71137e49: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 09:45:50.193706 26592 ts_tablet_manager.cc:1905] T 45097a1ff9ff4e4485104ad50a03f8f2 P c0742b25a62b4793bf20353f0d1f2a95: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 09:45:50.193819 26552 raft_consensus.cc:348] T 69856a9323ad48e4bd81faca5ce84fec P 7a79b23b92bc49bda09c1b6e71137e49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0742b25a62b4793bf20353f0d1f2a95" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 35609 } } peers { permanent_uuid: "49fa8338699749498ff4283df5548ab3" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 36055 } } peers { permanent_uuid: "7a79b23b92bc49bda09c1b6e71137e49" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 46139 } }
I20260812 09:45:50.193933 26552 raft_consensus.cc:374] T 69856a9323ad48e4bd81faca5ce84fec P 7a79b23b92bc49bda09c1b6e71137e49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:45:50.193966 26552 raft_consensus.cc:729] T 69856a9323ad48e4bd81faca5ce84fec P 7a79b23b92bc49bda09c1b6e71137e49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a79b23b92bc49bda09c1b6e71137e49, State: Initialized, Role: FOLLOWER
I20260812 09:45:50.194104 26552 consensus_queue.cc:260] T 69856a9323ad48e4bd81faca5ce84fec P 7a79b23b92bc49bda09c1b6e71137e49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0742b25a62b4793bf20353f0d1f2a95" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 35609 } } peers { permanent_uuid: "49fa8338699749498ff4283df5548ab3" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 36055 } } peers { permanent_uuid: "7a79b23b92bc49bda09c1b6e71137e49" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 46139 } }
I20260812 09:45:50.194381 26552 ts_tablet_manager.cc:1428] T 69856a9323ad48e4bd81faca5ce84fec P 7a79b23b92bc49bda09c1b6e71137e49: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260812 09:45:50.196452 26592 ts_tablet_manager.cc:1918] T 45097a1ff9ff4e4485104ad50a03f8f2 P c0742b25a62b4793bf20353f0d1f2a95: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 09:45:50.196584 26592 log.cc:1196] T 45097a1ff9ff4e4485104ad50a03f8f2 P c0742b25a62b4793bf20353f0d1f2a95: Deleting WAL directory at /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/minicluster-data/ts-0/wal/wals/45097a1ff9ff4e4485104ad50a03f8f2
I20260812 09:45:50.196962 26592 ts_tablet_manager.cc:1939] T 45097a1ff9ff4e4485104ad50a03f8f2 P c0742b25a62b4793bf20353f0d1f2a95: Deleting consensus metadata
I20260812 09:45:50.197258 26551 tablet_bootstrap.cc:492] T 69856a9323ad48e4bd81faca5ce84fec P c0742b25a62b4793bf20353f0d1f2a95: No bootstrap required, opened a new log
I20260812 09:45:50.197350 26551 ts_tablet_manager.cc:1397] T 69856a9323ad48e4bd81faca5ce84fec P c0742b25a62b4793bf20353f0d1f2a95: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20260812 09:45:50.197811 26551 raft_consensus.cc:348] T 69856a9323ad48e4bd81faca5ce84fec P c0742b25a62b4793bf20353f0d1f2a95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0742b25a62b4793bf20353f0d1f2a95" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 35609 } } peers { permanent_uuid: "49fa8338699749498ff4283df5548ab3" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 36055 } } peers { permanent_uuid: "7a79b23b92bc49bda09c1b6e71137e49" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 46139 } }
I20260812 09:45:50.197927 26551 raft_consensus.cc:374] T 69856a9323ad48e4bd81faca5ce84fec P c0742b25a62b4793bf20353f0d1f2a95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:45:50.197960 26551 raft_consensus.cc:729] T 69856a9323ad48e4bd81faca5ce84fec P c0742b25a62b4793bf20353f0d1f2a95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0742b25a62b4793bf20353f0d1f2a95, State: Initialized, Role: FOLLOWER
I20260812 09:45:50.198088 26551 consensus_queue.cc:260] T 69856a9323ad48e4bd81faca5ce84fec P c0742b25a62b4793bf20353f0d1f2a95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0742b25a62b4793bf20353f0d1f2a95" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 35609 } } peers { permanent_uuid: "49fa8338699749498ff4283df5548ab3" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 36055 } } peers { permanent_uuid: "7a79b23b92bc49bda09c1b6e71137e49" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 46139 } }
I20260812 09:45:50.198267 25702 catalog_manager.cc:4841] TS c0742b25a62b4793bf20353f0d1f2a95 (127.13.99.1:35609): tablet 45097a1ff9ff4e4485104ad50a03f8f2 (table test-table [id=db6c89950238450788907c13177cd49b]) successfully deleted
I20260812 09:45:50.198330 26551 ts_tablet_manager.cc:1428] T 69856a9323ad48e4bd81faca5ce84fec P c0742b25a62b4793bf20353f0d1f2a95: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20260812 09:45:50.208420 26500 tablet.cc:2369] T 69856a9323ad48e4bd81faca5ce84fec P 7a79b23b92bc49bda09c1b6e71137e49: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 09:45:50.225623 26248 tablet.cc:2369] T 69856a9323ad48e4bd81faca5ce84fec P 49fa8338699749498ff4283df5548ab3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 09:45:50.244300 26033 tablet.cc:2369] T 69856a9323ad48e4bd81faca5ce84fec P c0742b25a62b4793bf20353f0d1f2a95: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 09:45:50.364197 26561 raft_consensus.cc:482] T 69856a9323ad48e4bd81faca5ce84fec P 49fa8338699749498ff4283df5548ab3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 09:45:50.364415 26561 raft_consensus.cc:504] T 69856a9323ad48e4bd81faca5ce84fec P 49fa8338699749498ff4283df5548ab3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0742b25a62b4793bf20353f0d1f2a95" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 35609 } } peers { permanent_uuid: "49fa8338699749498ff4283df5548ab3" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 36055 } } peers { permanent_uuid: "7a79b23b92bc49bda09c1b6e71137e49" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 46139 } }
I20260812 09:45:50.365685 26561 leader_election.cc:288] T 69856a9323ad48e4bd81faca5ce84fec P 49fa8338699749498ff4283df5548ab3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0742b25a62b4793bf20353f0d1f2a95 (127.13.99.1:35609), 7a79b23b92bc49bda09c1b6e71137e49 (127.13.99.3:46139)
I20260812 09:45:50.371265 26543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:50.366364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.2:53321 (local address 127.13.99.3:46139)
0812 09:45:50.366530 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:50.366535 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:50.366553 (+ 18us) server_negotiation.cc:407] Connection header received
0812 09:45:50.366616 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:50.366619 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:50.366668 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:50.366758 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:45:50.367077 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:50.367852 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:50.369877 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:50.370504 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:50.370634 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:50.370706 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:50.371035 (+ 329us) server_negotiation.cc:299] Negotiation successful
0812 09:45:50.371128 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260812 09:45:50.371265 26539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:50.366048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.2:46681 (local address 127.13.99.1:35609)
0812 09:45:50.366210 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:50.366217 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:50.366305 (+ 88us) server_negotiation.cc:407] Connection header received
0812 09:45:50.366551 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:50.366554 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:50.366609 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:50.366700 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:45:50.367013 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:50.367839 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:50.369877 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:50.370504 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:50.370634 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:50.370710 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:50.371040 (+ 330us) server_negotiation.cc:299] Negotiation successful
0812 09:45:50.371140 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260812 09:45:50.371294 26598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:50.365918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.1:35609 (local address 127.13.99.2:46681)
0812 09:45:50.366206 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:50.366224 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:50.366410 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:50.366714 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:50.366717 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:50.366736 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:45:50.366897 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:50.366904 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:50.368003 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:50.368007 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:50.369690 (+ 1683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:50.369702 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:50.370665 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:50.370669 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:50.370824 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:50.370872 (+ 48us) client_negotiation.cc:769] Sending connection context
0812 09:45:50.370967 (+ 95us) client_negotiation.cc:240] Negotiation successful
0812 09:45:50.371100 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":251,"thread_start_us":90,"threads_started":1}
I20260812 09:45:50.371304 26600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:50.366123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.3:46139 (local address 127.13.99.2:53321)
0812 09:45:50.366403 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:50.366421 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:50.366528 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:50.366772 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:50.366775 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:50.366793 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:45:50.366935 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:50.366941 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:50.368003 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:50.368007 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:50.369693 (+ 1686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:50.369702 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:50.370665 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:50.370669 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:50.370821 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:50.370896 (+ 75us) client_negotiation.cc:769] Sending connection context
0812 09:45:50.370967 (+ 71us) client_negotiation.cc:240] Negotiation successful
0812 09:45:50.371100 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"mutex_wait_us":256,"thread_start_us":83,"threads_started":1}
I20260812 09:45:50.372056 26424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69856a9323ad48e4bd81faca5ce84fec" candidate_uuid: "49fa8338699749498ff4283df5548ab3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a79b23b92bc49bda09c1b6e71137e49" is_pre_election: true
I20260812 09:45:50.372057 25967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69856a9323ad48e4bd81faca5ce84fec" candidate_uuid: "49fa8338699749498ff4283df5548ab3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0742b25a62b4793bf20353f0d1f2a95" is_pre_election: true
I20260812 09:45:50.372236 26424 raft_consensus.cc:2454] T 69856a9323ad48e4bd81faca5ce84fec P 7a79b23b92bc49bda09c1b6e71137e49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49fa8338699749498ff4283df5548ab3 in term 0.
I20260812 09:45:50.372301 25967 raft_consensus.cc:2454] T 69856a9323ad48e4bd81faca5ce84fec P c0742b25a62b4793bf20353f0d1f2a95 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49fa8338699749498ff4283df5548ab3 in term 0.
I20260812 09:45:50.372675 26069 leader_election.cc:302] T 69856a9323ad48e4bd81faca5ce84fec P 49fa8338699749498ff4283df5548ab3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49fa8338699749498ff4283df5548ab3, 7a79b23b92bc49bda09c1b6e71137e49; no voters:
I20260812 09:45:50.372979 26561 raft_consensus.cc:2781] T 69856a9323ad48e4bd81faca5ce84fec P 49fa8338699749498ff4283df5548ab3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 09:45:50.373052 26561 raft_consensus.cc:482] T 69856a9323ad48e4bd81faca5ce84fec P 49fa8338699749498ff4283df5548ab3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 09:45:50.373092 26561 raft_consensus.cc:3037] T 69856a9323ad48e4bd81faca5ce84fec P 49fa8338699749498ff4283df5548ab3 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:45:50.374120 26561 raft_consensus.cc:504] T 69856a9323ad48e4bd81faca5ce84fec P 49fa8338699749498ff4283df5548ab3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0742b25a62b4793bf20353f0d1f2a95" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 35609 } } peers { permanent_uuid: "49fa8338699749498ff4283df5548ab3" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 36055 } } peers { permanent_uuid: "7a79b23b92bc49bda09c1b6e71137e49" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 46139 } }
I20260812 09:45:50.374471 26561 leader_election.cc:288] T 69856a9323ad48e4bd81faca5ce84fec P 49fa8338699749498ff4283df5548ab3 [CANDIDATE]: Term 1 election: Requested vote from peers c0742b25a62b4793bf20353f0d1f2a95 (127.13.99.1:35609), 7a79b23b92bc49bda09c1b6e71137e49 (127.13.99.3:46139)
I20260812 09:45:50.374856 25967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69856a9323ad48e4bd81faca5ce84fec" candidate_uuid: "49fa8338699749498ff4283df5548ab3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0742b25a62b4793bf20353f0d1f2a95"
I20260812 09:45:50.374856 26424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69856a9323ad48e4bd81faca5ce84fec" candidate_uuid: "49fa8338699749498ff4283df5548ab3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a79b23b92bc49bda09c1b6e71137e49"
I20260812 09:45:50.374984 26424 raft_consensus.cc:3037] T 69856a9323ad48e4bd81faca5ce84fec P 7a79b23b92bc49bda09c1b6e71137e49 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:45:50.374984 25967 raft_consensus.cc:3037] T 69856a9323ad48e4bd81faca5ce84fec P c0742b25a62b4793bf20353f0d1f2a95 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:45:50.376152 25967 raft_consensus.cc:2454] T 69856a9323ad48e4bd81faca5ce84fec P c0742b25a62b4793bf20353f0d1f2a95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49fa8338699749498ff4283df5548ab3 in term 1.
I20260812 09:45:50.376152 26424 raft_consensus.cc:2454] T 69856a9323ad48e4bd81faca5ce84fec P 7a79b23b92bc49bda09c1b6e71137e49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49fa8338699749498ff4283df5548ab3 in term 1.
I20260812 09:45:50.376430 26069 leader_election.cc:302] T 69856a9323ad48e4bd81faca5ce84fec P 49fa8338699749498ff4283df5548ab3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49fa8338699749498ff4283df5548ab3, 7a79b23b92bc49bda09c1b6e71137e49; no voters:
I20260812 09:45:50.376659 26561 raft_consensus.cc:2781] T 69856a9323ad48e4bd81faca5ce84fec P 49fa8338699749498ff4283df5548ab3 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 09:45:50.376838 26561 raft_consensus.cc:686] T 69856a9323ad48e4bd81faca5ce84fec P 49fa8338699749498ff4283df5548ab3 [term 1 LEADER]: Becoming Leader. State: Replica: 49fa8338699749498ff4283df5548ab3, State: Running, Role: LEADER
I20260812 09:45:50.377099 26561 consensus_queue.cc:237] T 69856a9323ad48e4bd81faca5ce84fec P 49fa8338699749498ff4283df5548ab3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0742b25a62b4793bf20353f0d1f2a95" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 35609 } } peers { permanent_uuid: "49fa8338699749498ff4283df5548ab3" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 36055 } } peers { permanent_uuid: "7a79b23b92bc49bda09c1b6e71137e49" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 46139 } }
I20260812 09:45:50.379485 25720 catalog_manager.cc:5495] T 69856a9323ad48e4bd81faca5ce84fec P 49fa8338699749498ff4283df5548ab3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49fa8338699749498ff4283df5548ab3 (127.13.99.2). New cstate: current_term: 1 leader_uuid: "49fa8338699749498ff4283df5548ab3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0742b25a62b4793bf20353f0d1f2a95" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 35609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49fa8338699749498ff4283df5548ab3" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 36055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a79b23b92bc49bda09c1b6e71137e49" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 46139 } health_report { overall_health: UNKNOWN } } }
Aug 12 09:45:50 dist-test-slave-jq4t krb5kdc[25661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527949, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.99.62@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM
I20260812 09:45:50.462654 26611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:50.454282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:35893 (local address 127.0.0.1:55138)
0812 09:45:50.454635 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:50.454651 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:50.454802 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:50.455064 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:50.455067 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:50.455497 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:45:50.455598 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:50.455605 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:50.456360 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:50.456365 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:50.456939 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:50.456956 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:50.457393 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:50.457396 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:50.457467 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:50.458035 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:45:50.458047 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:45:50.459514 (+ 1467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:45:50.461084 (+ 1570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:50.461093 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:50.461100 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:50.461323 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:50.461623 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:50.461628 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:50.461630 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:50.461711 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:50.462181 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:45:50.462187 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:45:50.462297 (+ 110us) client_negotiation.cc:769] Sending connection context
0812 09:45:50.462395 (+ 98us) client_negotiation.cc:240] Negotiation successful
0812 09:45:50.462519 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":114,"threads_started":1}
I20260812 09:45:50.462739 25803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:50.454466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55138 (local address 127.13.99.62:35893)
0812 09:45:50.454613 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:50.454618 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:50.454737 (+ 119us) server_negotiation.cc:407] Connection header received
0812 09:45:50.454904 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:50.454909 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:50.454952 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:50.455042 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:50.455727 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:50.456250 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:50.457068 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:50.457281 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:50.457391 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:50.459650 (+ 2259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:50.459656 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:50.459663 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:50.459683 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:50.460964 (+ 1281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:50.461426 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:50.461431 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:50.461433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:50.461531 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:50.461844 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:50.461848 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:50.461849 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:50.462041 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:50.462129 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:50.462457 (+ 328us) server_negotiation.cc:299] Negotiation successful
0812 09:45:50.462606 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":59}
I20260812 09:45:50.466559 25720 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.13.99.62@KRBTEST.COM'} at 127.0.0.1:55138:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260812 09:45:50.467334 25720 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 09:45:50.472219 25936 tablet_service.cc:1467] Processing CreateTablet for tablet 0d6f2039cc0e473ca2b86c8a83b30465 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c3c4d2d3f0a74a31b84c0735390b9616]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 09:45:50.472435 26140 tablet_service.cc:1467] Processing CreateTablet for tablet 0d6f2039cc0e473ca2b86c8a83b30465 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c3c4d2d3f0a74a31b84c0735390b9616]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 09:45:50.472510 26394 tablet_service.cc:1467] Processing CreateTablet for tablet 0d6f2039cc0e473ca2b86c8a83b30465 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c3c4d2d3f0a74a31b84c0735390b9616]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 09:45:50.472682 25936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d6f2039cc0e473ca2b86c8a83b30465. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:45:50.472860 26140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d6f2039cc0e473ca2b86c8a83b30465. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:45:50.472868 26394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d6f2039cc0e473ca2b86c8a83b30465. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:45:50.475558 26551 tablet_bootstrap.cc:492] T 0d6f2039cc0e473ca2b86c8a83b30465 P c0742b25a62b4793bf20353f0d1f2a95: Bootstrap starting.
I20260812 09:45:50.475569 26550 tablet_bootstrap.cc:492] T 0d6f2039cc0e473ca2b86c8a83b30465 P 49fa8338699749498ff4283df5548ab3: Bootstrap starting.
I20260812 09:45:50.475587 26552 tablet_bootstrap.cc:492] T 0d6f2039cc0e473ca2b86c8a83b30465 P 7a79b23b92bc49bda09c1b6e71137e49: Bootstrap starting.
I20260812 09:45:50.476441 26551 tablet_bootstrap.cc:654] T 0d6f2039cc0e473ca2b86c8a83b30465 P c0742b25a62b4793bf20353f0d1f2a95: Neither blocks nor log segments found. Creating new log.
I20260812 09:45:50.476516 26550 tablet_bootstrap.cc:654] T 0d6f2039cc0e473ca2b86c8a83b30465 P 49fa8338699749498ff4283df5548ab3: Neither blocks nor log segments found. Creating new log.
I20260812 09:45:50.476570 26552 tablet_bootstrap.cc:654] T 0d6f2039cc0e473ca2b86c8a83b30465 P 7a79b23b92bc49bda09c1b6e71137e49: Neither blocks nor log segments found. Creating new log.
I20260812 09:45:50.477764 26551 tablet_bootstrap.cc:492] T 0d6f2039cc0e473ca2b86c8a83b30465 P c0742b25a62b4793bf20353f0d1f2a95: No bootstrap required, opened a new log
I20260812 09:45:50.477860 26551 ts_tablet_manager.cc:1397] T 0d6f2039cc0e473ca2b86c8a83b30465 P c0742b25a62b4793bf20353f0d1f2a95: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260812 09:45:50.478222 26551 raft_consensus.cc:348] T 0d6f2039cc0e473ca2b86c8a83b30465 P c0742b25a62b4793bf20353f0d1f2a95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0742b25a62b4793bf20353f0d1f2a95" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 35609 } } peers { permanent_uuid: "49fa8338699749498ff4283df5548ab3" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 36055 } } peers { permanent_uuid: "7a79b23b92bc49bda09c1b6e71137e49" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 46139 } }
I20260812 09:45:50.478317 26551 raft_consensus.cc:374] T 0d6f2039cc0e473ca2b86c8a83b30465 P c0742b25a62b4793bf20353f0d1f2a95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:45:50.478339 26551 raft_consensus.cc:729] T 0d6f2039cc0e473ca2b86c8a83b30465 P c0742b25a62b4793bf20353f0d1f2a95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0742b25a62b4793bf20353f0d1f2a95, State: Initialized, Role: FOLLOWER
I20260812 09:45:50.478430 26551 consensus_queue.cc:260] T 0d6f2039cc0e473ca2b86c8a83b30465 P c0742b25a62b4793bf20353f0d1f2a95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0742b25a62b4793bf20353f0d1f2a95" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 35609 } } peers { permanent_uuid: "49fa8338699749498ff4283df5548ab3" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 36055 } } peers { permanent_uuid: "7a79b23b92bc49bda09c1b6e71137e49" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 46139 } }
I20260812 09:45:50.478704 26551 ts_tablet_manager.cc:1428] T 0d6f2039cc0e473ca2b86c8a83b30465 P c0742b25a62b4793bf20353f0d1f2a95: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260812 09:45:50.478650 26568 tablet_replica.cc:439] T 0d6f2039cc0e473ca2b86c8a83b30465 P c0742b25a62b4793bf20353f0d1f2a95: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0742b25a62b4793bf20353f0d1f2a95" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 35609 } } peers { permanent_uuid: "49fa8338699749498ff4283df5548ab3" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 36055 } } peers { permanent_uuid: "7a79b23b92bc49bda09c1b6e71137e49" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 46139 } } }
I20260812 09:45:50.478783 26568 tablet_replica.cc:442] T 0d6f2039cc0e473ca2b86c8a83b30465 P c0742b25a62b4793bf20353f0d1f2a95: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 09:45:50.479483 26550 tablet_bootstrap.cc:492] T 0d6f2039cc0e473ca2b86c8a83b30465 P 49fa8338699749498ff4283df5548ab3: No bootstrap required, opened a new log
I20260812 09:45:50.479575 26550 ts_tablet_manager.cc:1397] T 0d6f2039cc0e473ca2b86c8a83b30465 P 49fa8338699749498ff4283df5548ab3: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260812 09:45:50.479727 26552 tablet_bootstrap.cc:492] T 0d6f2039cc0e473ca2b86c8a83b30465 P 7a79b23b92bc49bda09c1b6e71137e49: No bootstrap required, opened a new log
I20260812 09:45:50.479826 26552 ts_tablet_manager.cc:1397] T 0d6f2039cc0e473ca2b86c8a83b30465 P 7a79b23b92bc49bda09c1b6e71137e49: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260812 09:45:50.480062 26550 raft_consensus.cc:348] T 0d6f2039cc0e473ca2b86c8a83b30465 P 49fa8338699749498ff4283df5548ab3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0742b25a62b4793bf20353f0d1f2a95" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 35609 } } peers { permanent_uuid: "49fa8338699749498ff4283df5548ab3" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 36055 } } peers { permanent_uuid: "7a79b23b92bc49bda09c1b6e71137e49" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 46139 } }
I20260812 09:45:50.480212 26550 raft_consensus.cc:374] T 0d6f2039cc0e473ca2b86c8a83b30465 P 49fa8338699749498ff4283df5548ab3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:45:50.480257 26550 raft_consensus.cc:729] T 0d6f2039cc0e473ca2b86c8a83b30465 P 49fa8338699749498ff4283df5548ab3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49fa8338699749498ff4283df5548ab3, State: Initialized, Role: FOLLOWER
I20260812 09:45:50.480270 26552 raft_consensus.cc:348] T 0d6f2039cc0e473ca2b86c8a83b30465 P 7a79b23b92bc49bda09c1b6e71137e49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0742b25a62b4793bf20353f0d1f2a95" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 35609 } } peers { permanent_uuid: "49fa8338699749498ff4283df5548ab3" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 36055 } } peers { permanent_uuid: "7a79b23b92bc49bda09c1b6e71137e49" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 46139 } }
I20260812 09:45:50.480391 26552 raft_consensus.cc:374] T 0d6f2039cc0e473ca2b86c8a83b30465 P 7a79b23b92bc49bda09c1b6e71137e49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:45:50.480422 26552 raft_consensus.cc:729] T 0d6f2039cc0e473ca2b86c8a83b30465 P 7a79b23b92bc49bda09c1b6e71137e49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a79b23b92bc49bda09c1b6e71137e49, State: Initialized, Role: FOLLOWER
I20260812 09:45:50.480403 26550 consensus_queue.cc:260] T 0d6f2039cc0e473ca2b86c8a83b30465 P 49fa8338699749498ff4283df5548ab3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0742b25a62b4793bf20353f0d1f2a95" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 35609 } } peers { permanent_uuid: "49fa8338699749498ff4283df5548ab3" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 36055 } } peers { permanent_uuid: "7a79b23b92bc49bda09c1b6e71137e49" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 46139 } }
I20260812 09:45:50.480605 26552 consensus_queue.cc:260] T 0d6f2039cc0e473ca2b86c8a83b30465 P 7a79b23b92bc49bda09c1b6e71137e49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0742b25a62b4793bf20353f0d1f2a95" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 35609 } } peers { permanent_uuid: "49fa8338699749498ff4283df5548ab3" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 36055 } } peers { permanent_uuid: "7a79b23b92bc49bda09c1b6e71137e49" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 46139 } }
I20260812 09:45:50.480676 26561 tablet_replica.cc:439] T 0d6f2039cc0e473ca2b86c8a83b30465 P 49fa8338699749498ff4283df5548ab3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0742b25a62b4793bf20353f0d1f2a95" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 35609 } } peers { permanent_uuid: "49fa8338699749498ff4283df5548ab3" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 36055 } } peers { permanent_uuid: "7a79b23b92bc49bda09c1b6e71137e49" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 46139 } } }
I20260812 09:45:50.480795 26561 tablet_replica.cc:442] T 0d6f2039cc0e473ca2b86c8a83b30465 P 49fa8338699749498ff4283df5548ab3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 09:45:50.480883 26552 ts_tablet_manager.cc:1428] T 0d6f2039cc0e473ca2b86c8a83b30465 P 7a79b23b92bc49bda09c1b6e71137e49: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260812 09:45:50.480989 26562 tablet_replica.cc:439] T 0d6f2039cc0e473ca2b86c8a83b30465 P 7a79b23b92bc49bda09c1b6e71137e49: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0742b25a62b4793bf20353f0d1f2a95" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 35609 } } peers { permanent_uuid: "49fa8338699749498ff4283df5548ab3" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 36055 } } peers { permanent_uuid: "7a79b23b92bc49bda09c1b6e71137e49" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 46139 } } }
I20260812 09:45:50.481158 26562 tablet_replica.cc:442] T 0d6f2039cc0e473ca2b86c8a83b30465 P 7a79b23b92bc49bda09c1b6e71137e49: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 09:45:50.482594 26550 ts_tablet_manager.cc:1428] T 0d6f2039cc0e473ca2b86c8a83b30465 P 49fa8338699749498ff4283df5548ab3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260812 09:45:50.500994 26562 raft_consensus.cc:482] T 0d6f2039cc0e473ca2b86c8a83b30465 P 7a79b23b92bc49bda09c1b6e71137e49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 09:45:50.501116 26562 raft_consensus.cc:504] T 0d6f2039cc0e473ca2b86c8a83b30465 P 7a79b23b92bc49bda09c1b6e71137e49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0742b25a62b4793bf20353f0d1f2a95" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 35609 } } peers { permanent_uuid: "49fa8338699749498ff4283df5548ab3" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 36055 } } peers { permanent_uuid: "7a79b23b92bc49bda09c1b6e71137e49" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 46139 } }
I20260812 09:45:50.502218 26562 leader_election.cc:288] T 0d6f2039cc0e473ca2b86c8a83b30465 P 7a79b23b92bc49bda09c1b6e71137e49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0742b25a62b4793bf20353f0d1f2a95 (127.13.99.1:35609), 49fa8338699749498ff4283df5548ab3 (127.13.99.2:36055)
I20260812 09:45:50.506620 26619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:50.502453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.2:36055 (local address 127.13.99.3:41773)
0812 09:45:50.503019 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:50.503032 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:50.503163 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:50.503498 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:50.503501 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:50.503516 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:45:50.503616 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:50.503621 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:50.504612 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:50.504615 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:50.505463 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:50.505472 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:50.506172 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:50.506176 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:50.506279 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:50.506306 (+ 27us) client_negotiation.cc:769] Sending connection context
0812 09:45:50.506389 (+ 83us) client_negotiation.cc:240] Negotiation successful
0812 09:45:50.506462 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":75,"threads_started":1}
I20260812 09:45:50.506623 26546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:50.502603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.3:41773 (local address 127.13.99.2:36055)
0812 09:45:50.502726 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:50.502731 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:50.503097 (+ 366us) server_negotiation.cc:407] Connection header received
0812 09:45:50.503276 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:50.503279 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:50.503322 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:50.503365 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:45:50.503734 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:50.504452 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:50.505599 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:50.506054 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:50.506162 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:50.506228 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:50.506410 (+ 182us) server_negotiation.cc:299] Negotiation successful
0812 09:45:50.506488 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260812 09:45:50.507129 26180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d6f2039cc0e473ca2b86c8a83b30465" candidate_uuid: "7a79b23b92bc49bda09c1b6e71137e49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49fa8338699749498ff4283df5548ab3" is_pre_election: true
I20260812 09:45:50.507361 26539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:50.502762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.3:39585 (local address 127.13.99.1:35609)
0812 09:45:50.502870 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:50.502875 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:50.502890 (+ 15us) server_negotiation.cc:407] Connection header received
0812 09:45:50.503034 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:50.503038 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:50.503095 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:50.503151 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:45:50.503600 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:50.504431 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:50.506008 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:50.506610 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:50.506720 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:50.506787 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:50.507156 (+ 369us) server_negotiation.cc:299] Negotiation successful
0812 09:45:50.507245 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260812 09:45:50.507272 26180 raft_consensus.cc:2454] T 0d6f2039cc0e473ca2b86c8a83b30465 P 49fa8338699749498ff4283df5548ab3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a79b23b92bc49bda09c1b6e71137e49 in term 0.
I20260812 09:45:50.507768 26306 leader_election.cc:302] T 0d6f2039cc0e473ca2b86c8a83b30465 P 7a79b23b92bc49bda09c1b6e71137e49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49fa8338699749498ff4283df5548ab3, 7a79b23b92bc49bda09c1b6e71137e49; no voters:
I20260812 09:45:50.507820 26618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:50.502422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.1:35609 (local address 127.13.99.3:39585)
0812 09:45:50.502762 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:50.502780 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:50.502880 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:50.503309 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:50.503312 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:50.503333 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:45:50.503466 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:50.503482 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:50.504556 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:50.504559 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:50.505873 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:50.505885 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:50.506914 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:50.506918 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:50.507017 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:50.507040 (+ 23us) client_negotiation.cc:769] Sending connection context
0812 09:45:50.507601 (+ 561us) client_negotiation.cc:240] Negotiation successful
0812 09:45:50.507680 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"mutex_wait_us":46,"thread_start_us":100,"threads_started":1}
I20260812 09:45:50.508085 26562 raft_consensus.cc:2781] T 0d6f2039cc0e473ca2b86c8a83b30465 P 7a79b23b92bc49bda09c1b6e71137e49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 09:45:50.508172 26562 raft_consensus.cc:482] T 0d6f2039cc0e473ca2b86c8a83b30465 P 7a79b23b92bc49bda09c1b6e71137e49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 09:45:50.508226 26562 raft_consensus.cc:3037] T 0d6f2039cc0e473ca2b86c8a83b30465 P 7a79b23b92bc49bda09c1b6e71137e49 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:45:50.509286 26562 raft_consensus.cc:504] T 0d6f2039cc0e473ca2b86c8a83b30465 P 7a79b23b92bc49bda09c1b6e71137e49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0742b25a62b4793bf20353f0d1f2a95" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 35609 } } peers { permanent_uuid: "49fa8338699749498ff4283df5548ab3" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 36055 } } peers { permanent_uuid: "7a79b23b92bc49bda09c1b6e71137e49" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 46139 } }
I20260812 09:45:50.509680 26562 leader_election.cc:288] T 0d6f2039cc0e473ca2b86c8a83b30465 P 7a79b23b92bc49bda09c1b6e71137e49 [CANDIDATE]: Term 1 election: Requested vote from peers c0742b25a62b4793bf20353f0d1f2a95 (127.13.99.1:35609), 49fa8338699749498ff4283df5548ab3 (127.13.99.2:36055)
I20260812 09:45:50.509881 25967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d6f2039cc0e473ca2b86c8a83b30465" candidate_uuid: "7a79b23b92bc49bda09c1b6e71137e49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0742b25a62b4793bf20353f0d1f2a95" is_pre_election: true
I20260812 09:45:50.510032 25967 raft_consensus.cc:2454] T 0d6f2039cc0e473ca2b86c8a83b30465 P c0742b25a62b4793bf20353f0d1f2a95 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a79b23b92bc49bda09c1b6e71137e49 in term 0.
I20260812 09:45:50.510069 26180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d6f2039cc0e473ca2b86c8a83b30465" candidate_uuid: "7a79b23b92bc49bda09c1b6e71137e49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49fa8338699749498ff4283df5548ab3"
I20260812 09:45:50.510006 25966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d6f2039cc0e473ca2b86c8a83b30465" candidate_uuid: "7a79b23b92bc49bda09c1b6e71137e49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0742b25a62b4793bf20353f0d1f2a95"
I20260812 09:45:50.510210 26180 raft_consensus.cc:3037] T 0d6f2039cc0e473ca2b86c8a83b30465 P 49fa8338699749498ff4283df5548ab3 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:45:50.510258 25966 raft_consensus.cc:3037] T 0d6f2039cc0e473ca2b86c8a83b30465 P c0742b25a62b4793bf20353f0d1f2a95 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:45:50.511309 25966 raft_consensus.cc:2454] T 0d6f2039cc0e473ca2b86c8a83b30465 P c0742b25a62b4793bf20353f0d1f2a95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a79b23b92bc49bda09c1b6e71137e49 in term 1.
I20260812 09:45:50.511431 26180 raft_consensus.cc:2454] T 0d6f2039cc0e473ca2b86c8a83b30465 P 49fa8338699749498ff4283df5548ab3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a79b23b92bc49bda09c1b6e71137e49 in term 1.
I20260812 09:45:50.511651 26307 leader_election.cc:302] T 0d6f2039cc0e473ca2b86c8a83b30465 P 7a79b23b92bc49bda09c1b6e71137e49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a79b23b92bc49bda09c1b6e71137e49, c0742b25a62b4793bf20353f0d1f2a95; no voters:
I20260812 09:45:50.511831 26562 raft_consensus.cc:2781] T 0d6f2039cc0e473ca2b86c8a83b30465 P 7a79b23b92bc49bda09c1b6e71137e49 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 09:45:50.512036 26562 raft_consensus.cc:686] T 0d6f2039cc0e473ca2b86c8a83b30465 P 7a79b23b92bc49bda09c1b6e71137e49 [term 1 LEADER]: Becoming Leader. State: Replica: 7a79b23b92bc49bda09c1b6e71137e49, State: Running, Role: LEADER
I20260812 09:45:50.512303 26562 consensus_queue.cc:237] T 0d6f2039cc0e473ca2b86c8a83b30465 P 7a79b23b92bc49bda09c1b6e71137e49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0742b25a62b4793bf20353f0d1f2a95" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 35609 } } peers { permanent_uuid: "49fa8338699749498ff4283df5548ab3" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 36055 } } peers { permanent_uuid: "7a79b23b92bc49bda09c1b6e71137e49" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 46139 } }
I20260812 09:45:50.513638 26620 tablet_replica.cc:439] T 0d6f2039cc0e473ca2b86c8a83b30465 P 7a79b23b92bc49bda09c1b6e71137e49: TxnStatusTablet state changed. Reason: New leader 7a79b23b92bc49bda09c1b6e71137e49. Latest consensus state: current_term: 1 leader_uuid: "7a79b23b92bc49bda09c1b6e71137e49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0742b25a62b4793bf20353f0d1f2a95" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 35609 } } peers { permanent_uuid: "49fa8338699749498ff4283df5548ab3" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 36055 } } peers { permanent_uuid: "7a79b23b92bc49bda09c1b6e71137e49" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 46139 } } }
I20260812 09:45:50.513762 26620 tablet_replica.cc:442] T 0d6f2039cc0e473ca2b86c8a83b30465 P 7a79b23b92bc49bda09c1b6e71137e49: This TxnStatusTablet replica's current role is: LEADER
I20260812 09:45:50.514240 26622 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260812 09:45:50.514398 26622 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260812 09:45:50.514931 25724 catalog_manager.cc:5495] T 0d6f2039cc0e473ca2b86c8a83b30465 P 7a79b23b92bc49bda09c1b6e71137e49 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a79b23b92bc49bda09c1b6e71137e49 (127.13.99.3). New cstate: current_term: 1 leader_uuid: "7a79b23b92bc49bda09c1b6e71137e49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0742b25a62b4793bf20353f0d1f2a95" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 35609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49fa8338699749498ff4283df5548ab3" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 36055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a79b23b92bc49bda09c1b6e71137e49" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 46139 } health_report { overall_health: HEALTHY } } }
I20260812 09:45:50.532059 26543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:50.528148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41676 (local address 127.13.99.3:46139)
0812 09:45:50.528297 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:50.528301 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:50.528317 (+ 16us) server_negotiation.cc:407] Connection header received
0812 09:45:50.528542 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:50.528549 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:50.528619 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:50.528736 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 09:45:50.529024 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:50.529639 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:50.530515 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:50.530751 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:50.530884 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:50.531197 (+ 313us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 09:45:50.531641 (+ 444us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 09:45:50.531771 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:50.531844 (+ 73us) server_negotiation.cc:299] Negotiation successful
0812 09:45:50.531901 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260812 09:45:50.532059 26611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:50.528042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.3:46139 (local address 127.0.0.1:41676)
0812 09:45:50.528184 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:50.528202 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:50.528319 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:50.528787 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:50.528790 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:50.528800 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 09:45:50.528904 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:50.528909 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:50.529750 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:50.529755 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:50.530371 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:50.530379 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:50.530884 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:50.530888 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:50.530984 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:50.531047 (+ 63us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 09:45:50.531743 (+ 696us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 09:45:50.531762 (+ 19us) client_negotiation.cc:769] Sending connection context
0812 09:45:50.531835 (+ 73us) client_negotiation.cc:240] Negotiation successful
0812 09:45:50.531930 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260812 09:45:50.535241 26180 raft_consensus.cc:1261] T 0d6f2039cc0e473ca2b86c8a83b30465 P 49fa8338699749498ff4283df5548ab3 [term 1 FOLLOWER]: Refusing update from remote peer 7a79b23b92bc49bda09c1b6e71137e49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 09:45:50.535451 25966 raft_consensus.cc:1261] T 0d6f2039cc0e473ca2b86c8a83b30465 P c0742b25a62b4793bf20353f0d1f2a95 [term 1 FOLLOWER]: Refusing update from remote peer 7a79b23b92bc49bda09c1b6e71137e49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 09:45:50.535770 26620 consensus_queue.cc:1035] T 0d6f2039cc0e473ca2b86c8a83b30465 P 7a79b23b92bc49bda09c1b6e71137e49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49fa8338699749498ff4283df5548ab3" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 36055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 09:45:50.535995 26562 consensus_queue.cc:1035] T 0d6f2039cc0e473ca2b86c8a83b30465 P 7a79b23b92bc49bda09c1b6e71137e49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0742b25a62b4793bf20353f0d1f2a95" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 35609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 09:45:50.537463 26561 tablet_replica.cc:439] T 0d6f2039cc0e473ca2b86c8a83b30465 P 49fa8338699749498ff4283df5548ab3: TxnStatusTablet state changed. Reason: New leader 7a79b23b92bc49bda09c1b6e71137e49. Latest consensus state: current_term: 1 leader_uuid: "7a79b23b92bc49bda09c1b6e71137e49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0742b25a62b4793bf20353f0d1f2a95" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 35609 } } peers { permanent_uuid: "49fa8338699749498ff4283df5548ab3" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 36055 } } peers { permanent_uuid: "7a79b23b92bc49bda09c1b6e71137e49" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 46139 } } }
I20260812 09:45:50.537573 26561 tablet_replica.cc:442] T 0d6f2039cc0e473ca2b86c8a83b30465 P 49fa8338699749498ff4283df5548ab3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 09:45:50.537752 26568 tablet_replica.cc:439] T 0d6f2039cc0e473ca2b86c8a83b30465 P c0742b25a62b4793bf20353f0d1f2a95: TxnStatusTablet state changed. Reason: New leader 7a79b23b92bc49bda09c1b6e71137e49. Latest consensus state: current_term: 1 leader_uuid: "7a79b23b92bc49bda09c1b6e71137e49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0742b25a62b4793bf20353f0d1f2a95" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 35609 } } peers { permanent_uuid: "49fa8338699749498ff4283df5548ab3" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 36055 } } peers { permanent_uuid: "7a79b23b92bc49bda09c1b6e71137e49" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 46139 } } }
I20260812 09:45:50.537875 26568 tablet_replica.cc:442] T 0d6f2039cc0e473ca2b86c8a83b30465 P c0742b25a62b4793bf20353f0d1f2a95: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 09:45:50.539547 26620 tablet_replica.cc:439] T 0d6f2039cc0e473ca2b86c8a83b30465 P 7a79b23b92bc49bda09c1b6e71137e49: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7a79b23b92bc49bda09c1b6e71137e49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0742b25a62b4793bf20353f0d1f2a95" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 35609 } } peers { permanent_uuid: "49fa8338699749498ff4283df5548ab3" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 36055 } } peers { permanent_uuid: "7a79b23b92bc49bda09c1b6e71137e49" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 46139 } } }
I20260812 09:45:50.539671 26620 tablet_replica.cc:442] T 0d6f2039cc0e473ca2b86c8a83b30465 P 7a79b23b92bc49bda09c1b6e71137e49: This TxnStatusTablet replica's current role is: LEADER
I20260812 09:45:50.539976 26620 tablet_replica.cc:439] T 0d6f2039cc0e473ca2b86c8a83b30465 P 7a79b23b92bc49bda09c1b6e71137e49: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7a79b23b92bc49bda09c1b6e71137e49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0742b25a62b4793bf20353f0d1f2a95" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 35609 } } peers { permanent_uuid: "49fa8338699749498ff4283df5548ab3" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 36055 } } peers { permanent_uuid: "7a79b23b92bc49bda09c1b6e71137e49" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 46139 } } }
I20260812 09:45:50.540081 26620 tablet_replica.cc:442] T 0d6f2039cc0e473ca2b86c8a83b30465 P 7a79b23b92bc49bda09c1b6e71137e49: This TxnStatusTablet replica's current role is: LEADER
I20260812 09:45:50.540794 26568 tablet_replica.cc:439] T 0d6f2039cc0e473ca2b86c8a83b30465 P c0742b25a62b4793bf20353f0d1f2a95: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7a79b23b92bc49bda09c1b6e71137e49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0742b25a62b4793bf20353f0d1f2a95" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 35609 } } peers { permanent_uuid: "49fa8338699749498ff4283df5548ab3" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 36055 } } peers { permanent_uuid: "7a79b23b92bc49bda09c1b6e71137e49" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 46139 } } }
I20260812 09:45:50.540910 26568 tablet_replica.cc:442] T 0d6f2039cc0e473ca2b86c8a83b30465 P c0742b25a62b4793bf20353f0d1f2a95: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 09:45:50.541288 26581 mvcc.cc:205] Tried to move back new op lower bound from 7317618485386960896 to 7317618485301350400. Current Snapshot: MvccSnapshot[applied={T|T < 7317618485386960896}]
I20260812 09:45:50.543481 26561 tablet_replica.cc:439] T 0d6f2039cc0e473ca2b86c8a83b30465 P 49fa8338699749498ff4283df5548ab3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7a79b23b92bc49bda09c1b6e71137e49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0742b25a62b4793bf20353f0d1f2a95" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 35609 } } peers { permanent_uuid: "49fa8338699749498ff4283df5548ab3" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 36055 } } peers { permanent_uuid: "7a79b23b92bc49bda09c1b6e71137e49" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 46139 } } }
I20260812 09:45:50.543593 26561 tablet_replica.cc:442] T 0d6f2039cc0e473ca2b86c8a83b30465 P 49fa8338699749498ff4283df5548ab3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 09:45:50.549705 26546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:50.545768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54628 (local address 127.13.99.2:36055)
0812 09:45:50.545877 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:50.545882 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:50.546005 (+ 123us) server_negotiation.cc:407] Connection header received
0812 09:45:50.546147 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:50.546150 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:50.546211 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:50.546321 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 09:45:50.546518 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:50.547301 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:50.548107 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:50.548445 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:50.548573 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:50.548812 (+ 239us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 09:45:50.549358 (+ 546us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 09:45:50.549470 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:50.549524 (+ 54us) server_negotiation.cc:299] Negotiation successful
0812 09:45:50.549578 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260812 09:45:50.560292 26543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:50.557022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.2:46839 (local address 127.13.99.3:46139)
0812 09:45:50.557165 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:50.557170 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:50.557298 (+ 128us) server_negotiation.cc:407] Connection header received
0812 09:45:50.557453 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:50.557456 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:50.557508 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:50.557587 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 09:45:50.557843 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:50.558348 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:50.559228 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:50.559455 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:50.559569 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:50.559860 (+ 291us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 09:45:50.559934 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 09:45:50.560020 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:50.560105 (+ 85us) server_negotiation.cc:299] Negotiation successful
0812 09:45:50.560155 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260812 09:45:50.560611 26634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:50.556915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.3:46139 (local address 127.13.99.2:46839)
0812 09:45:50.557215 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:50.557227 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:50.557351 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:50.557598 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:50.557601 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:50.557609 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 09:45:50.557737 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:50.557742 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:50.558455 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:50.558460 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:50.559107 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:50.559115 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:50.559584 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:50.559588 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:50.559681 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:50.559723 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 09:45:50.560017 (+ 294us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 09:45:50.560024 (+ 7us) client_negotiation.cc:769] Sending connection context
0812 09:45:50.560095 (+ 71us) client_negotiation.cc:240] Negotiation successful
0812 09:45:50.560170 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":103,"threads_started":1}
I20260812 09:45:50.568351 26424 raft_consensus.cc:1261] T 69856a9323ad48e4bd81faca5ce84fec P 7a79b23b92bc49bda09c1b6e71137e49 [term 1 FOLLOWER]: Refusing update from remote peer 49fa8338699749498ff4283df5548ab3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 09:45:50.568362 25966 raft_consensus.cc:1261] T 69856a9323ad48e4bd81faca5ce84fec P c0742b25a62b4793bf20353f0d1f2a95 [term 1 FOLLOWER]: Refusing update from remote peer 49fa8338699749498ff4283df5548ab3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 09:45:50.568848 26601 consensus_queue.cc:1035] T 69856a9323ad48e4bd81faca5ce84fec P 49fa8338699749498ff4283df5548ab3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a79b23b92bc49bda09c1b6e71137e49" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 46139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 09:45:50.569036 26561 consensus_queue.cc:1035] T 69856a9323ad48e4bd81faca5ce84fec P 49fa8338699749498ff4283df5548ab3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0742b25a62b4793bf20353f0d1f2a95" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 35609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 09:45:50.574350 26577 mvcc.cc:205] Tried to move back new op lower bound from 7317618485522878464 to 7317618484746575872. Current Snapshot: MvccSnapshot[applied={T|T < 7317618485522878464}]
I20260812 09:45:50.575933 26576 mvcc.cc:205] Tried to move back new op lower bound from 7317618485522878464 to 7317618484746575872. Current Snapshot: MvccSnapshot[applied={T|T < 7317618485522878464}]
I20260812 09:45:50.595618 26546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:50.591441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.3:58877 (local address 127.13.99.2:36055)
0812 09:45:50.591576 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:50.591581 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:50.591812 (+ 231us) server_negotiation.cc:407] Connection header received
0812 09:45:50.591949 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:50.591952 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:50.591994 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:50.592051 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 09:45:50.592385 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:50.593233 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:50.594499 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:50.594730 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:50.594817 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:50.595153 (+ 336us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 09:45:50.595222 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 09:45:50.595279 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:50.595396 (+ 117us) server_negotiation.cc:299] Negotiation successful
0812 09:45:50.595460 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260812 09:45:50.595664 26642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:50.591312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.2:36055 (local address 127.13.99.3:58877)
0812 09:45:50.591702 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:50.591719 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:50.591829 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:50.592090 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:50.592093 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:50.592103 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 09:45:50.592252 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:50.592259 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:50.593379 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:50.593382 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:50.594371 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:50.594383 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:50.594848 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:50.594852 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:50.594954 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:50.595013 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 09:45:50.595307 (+ 294us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 09:45:50.595314 (+ 7us) client_negotiation.cc:769] Sending connection context
0812 09:45:50.595383 (+ 69us) client_negotiation.cc:240] Negotiation successful
0812 09:45:50.595471 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":149,"threads_started":1}
I20260812 09:45:50.623958 25724 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55124:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 09:45:50.624162 25724 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55124:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 09:45:50.626626 25724 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7fb7f7ee6bd04ed7bb41ec750ae6d72a: Sending DeleteTablet for 3 replicas of tablet 69856a9323ad48e4bd81faca5ce84fec
I20260812 09:45:50.627192 25936 tablet_service.cc:1514] Processing DeleteTablet for tablet 69856a9323ad48e4bd81faca5ce84fec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 09:45:50 UTC) from {username='oryx', principal='oryx/127.13.99.62@KRBTEST.COM'} at 127.0.0.1:48072
I20260812 09:45:50.627403 26592 tablet_replica.cc:330] T 69856a9323ad48e4bd81faca5ce84fec P c0742b25a62b4793bf20353f0d1f2a95: stopping tablet replica
I20260812 09:45:50.627532 26592 raft_consensus.cc:2229] T 69856a9323ad48e4bd81faca5ce84fec P c0742b25a62b4793bf20353f0d1f2a95 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 09:45:50.627646 26592 raft_consensus.cc:2258] T 69856a9323ad48e4bd81faca5ce84fec P c0742b25a62b4793bf20353f0d1f2a95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 09:45:50.627965 26394 tablet_service.cc:1514] Processing DeleteTablet for tablet 69856a9323ad48e4bd81faca5ce84fec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 09:45:50 UTC) from {username='oryx', principal='oryx/127.13.99.62@KRBTEST.COM'} at 127.0.0.1:41666
I20260812 09:45:50.628150 26593 tablet_replica.cc:330] T 69856a9323ad48e4bd81faca5ce84fec P 7a79b23b92bc49bda09c1b6e71137e49: stopping tablet replica
I20260812 09:45:50.628266 26593 raft_consensus.cc:2229] T 69856a9323ad48e4bd81faca5ce84fec P 7a79b23b92bc49bda09c1b6e71137e49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 09:45:50.628386 26593 raft_consensus.cc:2258] T 69856a9323ad48e4bd81faca5ce84fec P 7a79b23b92bc49bda09c1b6e71137e49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 09:45:50.628670 26140 tablet_service.cc:1514] Processing DeleteTablet for tablet 69856a9323ad48e4bd81faca5ce84fec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 09:45:50 UTC) from {username='oryx', principal='oryx/127.13.99.62@KRBTEST.COM'} at 127.0.0.1:54622
I20260812 09:45:50.628865 26594 tablet_replica.cc:330] T 69856a9323ad48e4bd81faca5ce84fec P 49fa8338699749498ff4283df5548ab3: stopping tablet replica
I20260812 09:45:50.628984 26594 raft_consensus.cc:2229] T 69856a9323ad48e4bd81faca5ce84fec P 49fa8338699749498ff4283df5548ab3 [term 1 LEADER]: Raft consensus shutting down.
I20260812 09:45:50.629269 26593 ts_tablet_manager.cc:1905] T 69856a9323ad48e4bd81faca5ce84fec P 7a79b23b92bc49bda09c1b6e71137e49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 09:45:50.629299 26594 raft_consensus.cc:2258] T 69856a9323ad48e4bd81faca5ce84fec P 49fa8338699749498ff4283df5548ab3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 09:45:50.629371 26592 ts_tablet_manager.cc:1905] T 69856a9323ad48e4bd81faca5ce84fec P c0742b25a62b4793bf20353f0d1f2a95: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 09:45:50.630116 26594 ts_tablet_manager.cc:1905] T 69856a9323ad48e4bd81faca5ce84fec P 49fa8338699749498ff4283df5548ab3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 09:45:50.632110 26592 ts_tablet_manager.cc:1918] T 69856a9323ad48e4bd81faca5ce84fec P c0742b25a62b4793bf20353f0d1f2a95: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 09:45:50.632110 26593 ts_tablet_manager.cc:1918] T 69856a9323ad48e4bd81faca5ce84fec P 7a79b23b92bc49bda09c1b6e71137e49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 09:45:50.632184 26592 log.cc:1196] T 69856a9323ad48e4bd81faca5ce84fec P c0742b25a62b4793bf20353f0d1f2a95: Deleting WAL directory at /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/minicluster-data/ts-0/wal/wals/69856a9323ad48e4bd81faca5ce84fec
I20260812 09:45:50.632184 26593 log.cc:1196] T 69856a9323ad48e4bd81faca5ce84fec P 7a79b23b92bc49bda09c1b6e71137e49: Deleting WAL directory at /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/minicluster-data/ts-2/wal/wals/69856a9323ad48e4bd81faca5ce84fec
I20260812 09:45:50.632457 26593 ts_tablet_manager.cc:1939] T 69856a9323ad48e4bd81faca5ce84fec P 7a79b23b92bc49bda09c1b6e71137e49: Deleting consensus metadata
I20260812 09:45:50.632462 26592 ts_tablet_manager.cc:1939] T 69856a9323ad48e4bd81faca5ce84fec P c0742b25a62b4793bf20353f0d1f2a95: Deleting consensus metadata
I20260812 09:45:50.633389 25702 catalog_manager.cc:4841] TS c0742b25a62b4793bf20353f0d1f2a95 (127.13.99.1:35609): tablet 69856a9323ad48e4bd81faca5ce84fec (table test-table [id=2de3e16de8df4884b29bcc5f7fcca730]) successfully deleted
I20260812 09:45:50.633543 25701 catalog_manager.cc:4841] TS 7a79b23b92bc49bda09c1b6e71137e49 (127.13.99.3:46139): tablet 69856a9323ad48e4bd81faca5ce84fec (table test-table [id=2de3e16de8df4884b29bcc5f7fcca730]) successfully deleted
I20260812 09:45:50.637764 26594 ts_tablet_manager.cc:1918] T 69856a9323ad48e4bd81faca5ce84fec P 49fa8338699749498ff4283df5548ab3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 09:45:50.637928 26594 log.cc:1196] T 69856a9323ad48e4bd81faca5ce84fec P 49fa8338699749498ff4283df5548ab3: Deleting WAL directory at /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/minicluster-data/ts-1/wal/wals/69856a9323ad48e4bd81faca5ce84fec
I20260812 09:45:50.638273 26594 ts_tablet_manager.cc:1939] T 69856a9323ad48e4bd81faca5ce84fec P 49fa8338699749498ff4283df5548ab3: Deleting consensus metadata
I20260812 09:45:50.639120 25701 catalog_manager.cc:4841] TS 49fa8338699749498ff4283df5548ab3 (127.13.99.2:36055): tablet 69856a9323ad48e4bd81faca5ce84fec (table test-table [id=2de3e16de8df4884b29bcc5f7fcca730]) successfully deleted
Aug 12 09:45:50 dist-test-slave-jq4t krb5kdc[25661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527950, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 09:45:50 dist-test-slave-jq4t krb5kdc[25661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 09:45:50 dist-test-slave-jq4t krb5kdc[25661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
W20260812 09:45:50.672679 25803 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 09:45:50.664092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55146 (local address 127.13.99.62:35893)
0812 09:45:50.664391 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:50.664396 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:50.664619 (+ 223us) server_negotiation.cc:407] Connection header received
0812 09:45:50.664796 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:50.664800 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:50.664860 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:50.664952 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:50.665891 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:50.666795 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:50.668125 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:50.668573 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:50.668686 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:50.672543 (+ 3857us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55146: BlockingRecv error: recv got EOF from 127.0.0.1:55146 (error 108)
Metrics: {"server-negotiator.queue_time_us":186}
Aug 12 09:45:50 dist-test-slave-jq4t krb5kdc[25661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527950, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM
I20260812 09:45:50.685015 25803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:50.674927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55160 (local address 127.13.99.62:35893)
0812 09:45:50.675042 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:50.675046 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:50.675158 (+ 112us) server_negotiation.cc:407] Connection header received
0812 09:45:50.675295 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:50.675298 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:50.675345 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:50.675454 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:50.676136 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:50.676971 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:50.678137 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:50.678489 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:50.678599 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:50.681426 (+ 2827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:50.681430 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:50.681433 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:50.681462 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:50.683136 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:50.683618 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:50.683621 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:50.683623 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:50.683718 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:50.684016 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:50.684020 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:50.684022 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:50.684250 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:50.684397 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:50.684721 (+ 324us) server_negotiation.cc:299] Negotiation successful
0812 09:45:50.684845 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260812 09:45:50.688053 25724 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55160:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 09:45:50.688630 25724 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 09:45:50.694666 25936 tablet_service.cc:1467] Processing CreateTablet for tablet 57e95b5536374b8dabb9138f60fd3743 (DEFAULT_TABLE table=test-table [id=683dc55eaf544b59a9c22c4bd9f05db7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 09:45:50.694980 25936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57e95b5536374b8dabb9138f60fd3743. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:45:50.696768 26394 tablet_service.cc:1467] Processing CreateTablet for tablet 57e95b5536374b8dabb9138f60fd3743 (DEFAULT_TABLE table=test-table [id=683dc55eaf544b59a9c22c4bd9f05db7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 09:45:50.697072 26394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57e95b5536374b8dabb9138f60fd3743. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:45:50.699520 26552 tablet_bootstrap.cc:492] T 57e95b5536374b8dabb9138f60fd3743 P 7a79b23b92bc49bda09c1b6e71137e49: Bootstrap starting.
I20260812 09:45:50.700347 26552 tablet_bootstrap.cc:654] T 57e95b5536374b8dabb9138f60fd3743 P 7a79b23b92bc49bda09c1b6e71137e49: Neither blocks nor log segments found. Creating new log.
I20260812 09:45:50.701526 26140 tablet_service.cc:1467] Processing CreateTablet for tablet 57e95b5536374b8dabb9138f60fd3743 (DEFAULT_TABLE table=test-table [id=683dc55eaf544b59a9c22c4bd9f05db7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 09:45:50.701802 26140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57e95b5536374b8dabb9138f60fd3743. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:45:50.704234 26550 tablet_bootstrap.cc:492] T 57e95b5536374b8dabb9138f60fd3743 P 49fa8338699749498ff4283df5548ab3: Bootstrap starting.
I20260812 09:45:50.705183 26550 tablet_bootstrap.cc:654] T 57e95b5536374b8dabb9138f60fd3743 P 49fa8338699749498ff4283df5548ab3: Neither blocks nor log segments found. Creating new log.
I20260812 09:45:50.705466 26552 tablet_bootstrap.cc:492] T 57e95b5536374b8dabb9138f60fd3743 P 7a79b23b92bc49bda09c1b6e71137e49: No bootstrap required, opened a new log
I20260812 09:45:50.705547 26552 ts_tablet_manager.cc:1397] T 57e95b5536374b8dabb9138f60fd3743 P 7a79b23b92bc49bda09c1b6e71137e49: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20260812 09:45:50.706135 26552 raft_consensus.cc:348] T 57e95b5536374b8dabb9138f60fd3743 P 7a79b23b92bc49bda09c1b6e71137e49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0742b25a62b4793bf20353f0d1f2a95" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 35609 } } peers { permanent_uuid: "49fa8338699749498ff4283df5548ab3" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 36055 } } peers { permanent_uuid: "7a79b23b92bc49bda09c1b6e71137e49" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 46139 } }
I20260812 09:45:50.706265 26552 raft_consensus.cc:374] T 57e95b5536374b8dabb9138f60fd3743 P 7a79b23b92bc49bda09c1b6e71137e49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:45:50.706298 26552 raft_consensus.cc:729] T 57e95b5536374b8dabb9138f60fd3743 P 7a79b23b92bc49bda09c1b6e71137e49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a79b23b92bc49bda09c1b6e71137e49, State: Initialized, Role: FOLLOWER
I20260812 09:45:50.706532 26552 consensus_queue.cc:260] T 57e95b5536374b8dabb9138f60fd3743 P 7a79b23b92bc49bda09c1b6e71137e49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0742b25a62b4793bf20353f0d1f2a95" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 35609 } } peers { permanent_uuid: "49fa8338699749498ff4283df5548ab3" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 36055 } } peers { permanent_uuid: "7a79b23b92bc49bda09c1b6e71137e49" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 46139 } }
I20260812 09:45:50.707895 26551 tablet_bootstrap.cc:492] T 57e95b5536374b8dabb9138f60fd3743 P c0742b25a62b4793bf20353f0d1f2a95: Bootstrap starting.
I20260812 09:45:50.708930 26551 tablet_bootstrap.cc:654] T 57e95b5536374b8dabb9138f60fd3743 P c0742b25a62b4793bf20353f0d1f2a95: Neither blocks nor log segments found. Creating new log.
I20260812 09:45:50.709141 26552 ts_tablet_manager.cc:1428] T 57e95b5536374b8dabb9138f60fd3743 P 7a79b23b92bc49bda09c1b6e71137e49: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260812 09:45:50.709931 26550 tablet_bootstrap.cc:492] T 57e95b5536374b8dabb9138f60fd3743 P 49fa8338699749498ff4283df5548ab3: No bootstrap required, opened a new log
I20260812 09:45:50.710024 26550 ts_tablet_manager.cc:1397] T 57e95b5536374b8dabb9138f60fd3743 P 49fa8338699749498ff4283df5548ab3: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260812 09:45:50.710532 26550 raft_consensus.cc:348] T 57e95b5536374b8dabb9138f60fd3743 P 49fa8338699749498ff4283df5548ab3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0742b25a62b4793bf20353f0d1f2a95" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 35609 } } peers { permanent_uuid: "49fa8338699749498ff4283df5548ab3" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 36055 } } peers { permanent_uuid: "7a79b23b92bc49bda09c1b6e71137e49" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 46139 } }
I20260812 09:45:50.710669 26550 raft_consensus.cc:374] T 57e95b5536374b8dabb9138f60fd3743 P 49fa8338699749498ff4283df5548ab3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:45:50.710714 26550 raft_consensus.cc:729] T 57e95b5536374b8dabb9138f60fd3743 P 49fa8338699749498ff4283df5548ab3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49fa8338699749498ff4283df5548ab3, State: Initialized, Role: FOLLOWER
I20260812 09:45:50.710914 26551 tablet_bootstrap.cc:492] T 57e95b5536374b8dabb9138f60fd3743 P c0742b25a62b4793bf20353f0d1f2a95: No bootstrap required, opened a new log
I20260812 09:45:50.710876 26550 consensus_queue.cc:260] T 57e95b5536374b8dabb9138f60fd3743 P 49fa8338699749498ff4283df5548ab3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0742b25a62b4793bf20353f0d1f2a95" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 35609 } } peers { permanent_uuid: "49fa8338699749498ff4283df5548ab3" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 36055 } } peers { permanent_uuid: "7a79b23b92bc49bda09c1b6e71137e49" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 46139 } }
I20260812 09:45:50.710990 26551 ts_tablet_manager.cc:1397] T 57e95b5536374b8dabb9138f60fd3743 P c0742b25a62b4793bf20353f0d1f2a95: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260812 09:45:50.711159 26550 ts_tablet_manager.cc:1428] T 57e95b5536374b8dabb9138f60fd3743 P 49fa8338699749498ff4283df5548ab3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 09:45:50.711426 26551 raft_consensus.cc:348] T 57e95b5536374b8dabb9138f60fd3743 P c0742b25a62b4793bf20353f0d1f2a95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0742b25a62b4793bf20353f0d1f2a95" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 35609 } } peers { permanent_uuid: "49fa8338699749498ff4283df5548ab3" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 36055 } } peers { permanent_uuid: "7a79b23b92bc49bda09c1b6e71137e49" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 46139 } }
I20260812 09:45:50.711581 26551 raft_consensus.cc:374] T 57e95b5536374b8dabb9138f60fd3743 P c0742b25a62b4793bf20353f0d1f2a95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:45:50.711620 26551 raft_consensus.cc:729] T 57e95b5536374b8dabb9138f60fd3743 P c0742b25a62b4793bf20353f0d1f2a95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0742b25a62b4793bf20353f0d1f2a95, State: Initialized, Role: FOLLOWER
I20260812 09:45:50.711761 26551 consensus_queue.cc:260] T 57e95b5536374b8dabb9138f60fd3743 P c0742b25a62b4793bf20353f0d1f2a95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0742b25a62b4793bf20353f0d1f2a95" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 35609 } } peers { permanent_uuid: "49fa8338699749498ff4283df5548ab3" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 36055 } } peers { permanent_uuid: "7a79b23b92bc49bda09c1b6e71137e49" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 46139 } }
I20260812 09:45:50.712059 26551 ts_tablet_manager.cc:1428] T 57e95b5536374b8dabb9138f60fd3743 P c0742b25a62b4793bf20353f0d1f2a95: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 09:45:50.868945 26635 raft_consensus.cc:482] T 57e95b5536374b8dabb9138f60fd3743 P 7a79b23b92bc49bda09c1b6e71137e49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 09:45:50.869087 26635 raft_consensus.cc:504] T 57e95b5536374b8dabb9138f60fd3743 P 7a79b23b92bc49bda09c1b6e71137e49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0742b25a62b4793bf20353f0d1f2a95" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 35609 } } peers { permanent_uuid: "49fa8338699749498ff4283df5548ab3" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 36055 } } peers { permanent_uuid: "7a79b23b92bc49bda09c1b6e71137e49" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 46139 } }
I20260812 09:45:50.869830 26635 leader_election.cc:288] T 57e95b5536374b8dabb9138f60fd3743 P 7a79b23b92bc49bda09c1b6e71137e49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0742b25a62b4793bf20353f0d1f2a95 (127.13.99.1:35609), 49fa8338699749498ff4283df5548ab3 (127.13.99.2:36055)
I20260812 09:45:50.870325 26180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57e95b5536374b8dabb9138f60fd3743" candidate_uuid: "7a79b23b92bc49bda09c1b6e71137e49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49fa8338699749498ff4283df5548ab3" is_pre_election: true
I20260812 09:45:50.870340 25966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57e95b5536374b8dabb9138f60fd3743" candidate_uuid: "7a79b23b92bc49bda09c1b6e71137e49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0742b25a62b4793bf20353f0d1f2a95" is_pre_election: true
I20260812 09:45:50.870512 26180 raft_consensus.cc:2454] T 57e95b5536374b8dabb9138f60fd3743 P 49fa8338699749498ff4283df5548ab3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a79b23b92bc49bda09c1b6e71137e49 in term 0.
I20260812 09:45:50.870522 25966 raft_consensus.cc:2454] T 57e95b5536374b8dabb9138f60fd3743 P c0742b25a62b4793bf20353f0d1f2a95 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a79b23b92bc49bda09c1b6e71137e49 in term 0.
I20260812 09:45:50.870893 26306 leader_election.cc:302] T 57e95b5536374b8dabb9138f60fd3743 P 7a79b23b92bc49bda09c1b6e71137e49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49fa8338699749498ff4283df5548ab3, 7a79b23b92bc49bda09c1b6e71137e49; no voters:
I20260812 09:45:50.871102 26635 raft_consensus.cc:2781] T 57e95b5536374b8dabb9138f60fd3743 P 7a79b23b92bc49bda09c1b6e71137e49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 09:45:50.871167 26635 raft_consensus.cc:482] T 57e95b5536374b8dabb9138f60fd3743 P 7a79b23b92bc49bda09c1b6e71137e49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 09:45:50.871212 26635 raft_consensus.cc:3037] T 57e95b5536374b8dabb9138f60fd3743 P 7a79b23b92bc49bda09c1b6e71137e49 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:45:50.872503 26635 raft_consensus.cc:504] T 57e95b5536374b8dabb9138f60fd3743 P 7a79b23b92bc49bda09c1b6e71137e49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0742b25a62b4793bf20353f0d1f2a95" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 35609 } } peers { permanent_uuid: "49fa8338699749498ff4283df5548ab3" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 36055 } } peers { permanent_uuid: "7a79b23b92bc49bda09c1b6e71137e49" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 46139 } }
I20260812 09:45:50.872903 26635 leader_election.cc:288] T 57e95b5536374b8dabb9138f60fd3743 P 7a79b23b92bc49bda09c1b6e71137e49 [CANDIDATE]: Term 1 election: Requested vote from peers c0742b25a62b4793bf20353f0d1f2a95 (127.13.99.1:35609), 49fa8338699749498ff4283df5548ab3 (127.13.99.2:36055)
I20260812 09:45:50.873273 26180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57e95b5536374b8dabb9138f60fd3743" candidate_uuid: "7a79b23b92bc49bda09c1b6e71137e49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49fa8338699749498ff4283df5548ab3"
I20260812 09:45:50.873375 26180 raft_consensus.cc:3037] T 57e95b5536374b8dabb9138f60fd3743 P 49fa8338699749498ff4283df5548ab3 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:45:50.873327 25966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57e95b5536374b8dabb9138f60fd3743" candidate_uuid: "7a79b23b92bc49bda09c1b6e71137e49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0742b25a62b4793bf20353f0d1f2a95"
I20260812 09:45:50.873502 25966 raft_consensus.cc:3037] T 57e95b5536374b8dabb9138f60fd3743 P c0742b25a62b4793bf20353f0d1f2a95 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:45:50.874545 26180 raft_consensus.cc:2454] T 57e95b5536374b8dabb9138f60fd3743 P 49fa8338699749498ff4283df5548ab3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a79b23b92bc49bda09c1b6e71137e49 in term 1.
I20260812 09:45:50.874567 25966 raft_consensus.cc:2454] T 57e95b5536374b8dabb9138f60fd3743 P c0742b25a62b4793bf20353f0d1f2a95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a79b23b92bc49bda09c1b6e71137e49 in term 1.
I20260812 09:45:50.874827 26306 leader_election.cc:302] T 57e95b5536374b8dabb9138f60fd3743 P 7a79b23b92bc49bda09c1b6e71137e49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49fa8338699749498ff4283df5548ab3, 7a79b23b92bc49bda09c1b6e71137e49; no voters:
I20260812 09:45:50.874992 26635 raft_consensus.cc:2781] T 57e95b5536374b8dabb9138f60fd3743 P 7a79b23b92bc49bda09c1b6e71137e49 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 09:45:50.875108 26635 raft_consensus.cc:686] T 57e95b5536374b8dabb9138f60fd3743 P 7a79b23b92bc49bda09c1b6e71137e49 [term 1 LEADER]: Becoming Leader. State: Replica: 7a79b23b92bc49bda09c1b6e71137e49, State: Running, Role: LEADER
I20260812 09:45:50.875267 26635 consensus_queue.cc:237] T 57e95b5536374b8dabb9138f60fd3743 P 7a79b23b92bc49bda09c1b6e71137e49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0742b25a62b4793bf20353f0d1f2a95" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 35609 } } peers { permanent_uuid: "49fa8338699749498ff4283df5548ab3" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 36055 } } peers { permanent_uuid: "7a79b23b92bc49bda09c1b6e71137e49" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 46139 } }
I20260812 09:45:50.876964 25725 catalog_manager.cc:5495] T 57e95b5536374b8dabb9138f60fd3743 P 7a79b23b92bc49bda09c1b6e71137e49 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a79b23b92bc49bda09c1b6e71137e49 (127.13.99.3). New cstate: current_term: 1 leader_uuid: "7a79b23b92bc49bda09c1b6e71137e49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0742b25a62b4793bf20353f0d1f2a95" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 35609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49fa8338699749498ff4283df5548ab3" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 36055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a79b23b92bc49bda09c1b6e71137e49" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 46139 } health_report { overall_health: HEALTHY } } }
I20260812 09:45:50.891136 26543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:50.887826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41688 (local address 127.13.99.3:46139)
0812 09:45:50.887961 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:50.887968 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:50.887982 (+ 14us) server_negotiation.cc:407] Connection header received
0812 09:45:50.888187 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:50.888191 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:50.888244 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:50.888340 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 09:45:50.888637 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:50.889147 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:50.890014 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:50.890243 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:50.890368 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:50.890649 (+ 281us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 09:45:50.890720 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 09:45:50.890814 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:50.890940 (+ 126us) server_negotiation.cc:299] Negotiation successful
0812 09:45:50.890994 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260812 09:45:50.893509 26180 raft_consensus.cc:1261] T 57e95b5536374b8dabb9138f60fd3743 P 49fa8338699749498ff4283df5548ab3 [term 1 FOLLOWER]: Refusing update from remote peer 7a79b23b92bc49bda09c1b6e71137e49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 09:45:50.893520 25966 raft_consensus.cc:1261] T 57e95b5536374b8dabb9138f60fd3743 P c0742b25a62b4793bf20353f0d1f2a95 [term 1 FOLLOWER]: Refusing update from remote peer 7a79b23b92bc49bda09c1b6e71137e49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 09:45:50.894024 26635 consensus_queue.cc:1035] T 57e95b5536374b8dabb9138f60fd3743 P 7a79b23b92bc49bda09c1b6e71137e49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0742b25a62b4793bf20353f0d1f2a95" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 35609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 09:45:50.894192 26620 consensus_queue.cc:1035] T 57e95b5536374b8dabb9138f60fd3743 P 7a79b23b92bc49bda09c1b6e71137e49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49fa8338699749498ff4283df5548ab3" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 36055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 09:45:50.903709 25720 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55160:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 09:45:50.903858 25720 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55160:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 09:45:50.906009 25720 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7fb7f7ee6bd04ed7bb41ec750ae6d72a: Sending DeleteTablet for 3 replicas of tablet 57e95b5536374b8dabb9138f60fd3743
I20260812 09:45:50.906674 25936 tablet_service.cc:1514] Processing DeleteTablet for tablet 57e95b5536374b8dabb9138f60fd3743 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 09:45:50 UTC) from {username='oryx', principal='oryx/127.13.99.62@KRBTEST.COM'} at 127.0.0.1:48072
I20260812 09:45:50.906663 26394 tablet_service.cc:1514] Processing DeleteTablet for tablet 57e95b5536374b8dabb9138f60fd3743 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 09:45:50 UTC) from {username='oryx', principal='oryx/127.13.99.62@KRBTEST.COM'} at 127.0.0.1:41666
I20260812 09:45:50.906839 26593 tablet_replica.cc:330] T 57e95b5536374b8dabb9138f60fd3743 P 7a79b23b92bc49bda09c1b6e71137e49: stopping tablet replica
I20260812 09:45:50.906848 26140 tablet_service.cc:1514] Processing DeleteTablet for tablet 57e95b5536374b8dabb9138f60fd3743 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 09:45:50 UTC) from {username='oryx', principal='oryx/127.13.99.62@KRBTEST.COM'} at 127.0.0.1:54622
I20260812 09:45:50.906968 26593 raft_consensus.cc:2229] T 57e95b5536374b8dabb9138f60fd3743 P 7a79b23b92bc49bda09c1b6e71137e49 [term 1 LEADER]: Raft consensus shutting down.
I20260812 09:45:50.907027 26592 tablet_replica.cc:330] T 57e95b5536374b8dabb9138f60fd3743 P c0742b25a62b4793bf20353f0d1f2a95: stopping tablet replica
I20260812 09:45:50.907146 26592 raft_consensus.cc:2229] T 57e95b5536374b8dabb9138f60fd3743 P c0742b25a62b4793bf20353f0d1f2a95 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 09:45:50.907267 26592 raft_consensus.cc:2258] T 57e95b5536374b8dabb9138f60fd3743 P c0742b25a62b4793bf20353f0d1f2a95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 09:45:50.907267 26593 raft_consensus.cc:2258] T 57e95b5536374b8dabb9138f60fd3743 P 7a79b23b92bc49bda09c1b6e71137e49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 09:45:50.907438 26594 tablet_replica.cc:330] T 57e95b5536374b8dabb9138f60fd3743 P 49fa8338699749498ff4283df5548ab3: stopping tablet replica
I20260812 09:45:50.907557 26594 raft_consensus.cc:2229] T 57e95b5536374b8dabb9138f60fd3743 P 49fa8338699749498ff4283df5548ab3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 09:45:50.907680 26594 raft_consensus.cc:2258] T 57e95b5536374b8dabb9138f60fd3743 P 49fa8338699749498ff4283df5548ab3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 09:45:50.907915 26593 ts_tablet_manager.cc:1905] T 57e95b5536374b8dabb9138f60fd3743 P 7a79b23b92bc49bda09c1b6e71137e49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 09:45:50.908468 26592 ts_tablet_manager.cc:1905] T 57e95b5536374b8dabb9138f60fd3743 P c0742b25a62b4793bf20353f0d1f2a95: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 09:45:50.908787 25720 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55160:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 09:45:50.909148 25720 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 09:45:50.910382 26593 ts_tablet_manager.cc:1918] T 57e95b5536374b8dabb9138f60fd3743 P 7a79b23b92bc49bda09c1b6e71137e49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 09:45:50.910437 26593 log.cc:1196] T 57e95b5536374b8dabb9138f60fd3743 P 7a79b23b92bc49bda09c1b6e71137e49: Deleting WAL directory at /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/minicluster-data/ts-2/wal/wals/57e95b5536374b8dabb9138f60fd3743
I20260812 09:45:50.910626 26593 ts_tablet_manager.cc:1939] T 57e95b5536374b8dabb9138f60fd3743 P 7a79b23b92bc49bda09c1b6e71137e49: Deleting consensus metadata
I20260812 09:45:50.910950 26592 ts_tablet_manager.cc:1918] T 57e95b5536374b8dabb9138f60fd3743 P c0742b25a62b4793bf20353f0d1f2a95: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 09:45:50.911026 26592 log.cc:1196] T 57e95b5536374b8dabb9138f60fd3743 P c0742b25a62b4793bf20353f0d1f2a95: Deleting WAL directory at /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/minicluster-data/ts-0/wal/wals/57e95b5536374b8dabb9138f60fd3743
I20260812 09:45:50.911284 26592 ts_tablet_manager.cc:1939] T 57e95b5536374b8dabb9138f60fd3743 P c0742b25a62b4793bf20353f0d1f2a95: Deleting consensus metadata
I20260812 09:45:50.911384 25701 catalog_manager.cc:4841] TS 7a79b23b92bc49bda09c1b6e71137e49 (127.13.99.3:46139): tablet 57e95b5536374b8dabb9138f60fd3743 (table test-table [id=683dc55eaf544b59a9c22c4bd9f05db7]) successfully deleted
I20260812 09:45:50.911890 25702 catalog_manager.cc:4841] TS c0742b25a62b4793bf20353f0d1f2a95 (127.13.99.1:35609): tablet 57e95b5536374b8dabb9138f60fd3743 (table test-table [id=683dc55eaf544b59a9c22c4bd9f05db7]) successfully deleted
I20260812 09:45:50.912221 26594 ts_tablet_manager.cc:1905] T 57e95b5536374b8dabb9138f60fd3743 P 49fa8338699749498ff4283df5548ab3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 09:45:50.914680 26594 ts_tablet_manager.cc:1918] T 57e95b5536374b8dabb9138f60fd3743 P 49fa8338699749498ff4283df5548ab3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 09:45:50.914765 26594 log.cc:1196] T 57e95b5536374b8dabb9138f60fd3743 P 49fa8338699749498ff4283df5548ab3: Deleting WAL directory at /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/minicluster-data/ts-1/wal/wals/57e95b5536374b8dabb9138f60fd3743
I20260812 09:45:50.915048 26594 ts_tablet_manager.cc:1939] T 57e95b5536374b8dabb9138f60fd3743 P 49fa8338699749498ff4283df5548ab3: Deleting consensus metadata
I20260812 09:45:50.915038 25936 tablet_service.cc:1467] Processing CreateTablet for tablet 2e737064c8e34096976e90d864bc9783 (DEFAULT_TABLE table=test-table [id=d7905afda122460392dd1ef0bbd29b06]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 09:45:50.915176 26140 tablet_service.cc:1467] Processing CreateTablet for tablet 2e737064c8e34096976e90d864bc9783 (DEFAULT_TABLE table=test-table [id=d7905afda122460392dd1ef0bbd29b06]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 09:45:50.915277 25936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e737064c8e34096976e90d864bc9783. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:45:50.915390 26140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e737064c8e34096976e90d864bc9783. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:45:50.917143 26550 tablet_bootstrap.cc:492] T 2e737064c8e34096976e90d864bc9783 P 49fa8338699749498ff4283df5548ab3: Bootstrap starting.
I20260812 09:45:50.917479 26551 tablet_bootstrap.cc:492] T 2e737064c8e34096976e90d864bc9783 P c0742b25a62b4793bf20353f0d1f2a95: Bootstrap starting.
I20260812 09:45:50.917472 25701 catalog_manager.cc:4841] TS 49fa8338699749498ff4283df5548ab3 (127.13.99.2:36055): tablet 57e95b5536374b8dabb9138f60fd3743 (table test-table [id=683dc55eaf544b59a9c22c4bd9f05db7]) successfully deleted
I20260812 09:45:50.917971 26550 tablet_bootstrap.cc:654] T 2e737064c8e34096976e90d864bc9783 P 49fa8338699749498ff4283df5548ab3: Neither blocks nor log segments found. Creating new log.
I20260812 09:45:50.918247 26551 tablet_bootstrap.cc:654] T 2e737064c8e34096976e90d864bc9783 P c0742b25a62b4793bf20353f0d1f2a95: Neither blocks nor log segments found. Creating new log.
I20260812 09:45:50.918612 26394 tablet_service.cc:1467] Processing CreateTablet for tablet 2e737064c8e34096976e90d864bc9783 (DEFAULT_TABLE table=test-table [id=d7905afda122460392dd1ef0bbd29b06]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 09:45:50.918884 26394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e737064c8e34096976e90d864bc9783. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:45:50.919448 26551 tablet_bootstrap.cc:492] T 2e737064c8e34096976e90d864bc9783 P c0742b25a62b4793bf20353f0d1f2a95: No bootstrap required, opened a new log
I20260812 09:45:50.919530 26551 ts_tablet_manager.cc:1397] T 2e737064c8e34096976e90d864bc9783 P c0742b25a62b4793bf20353f0d1f2a95: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 09:45:50.919885 26551 raft_consensus.cc:348] T 2e737064c8e34096976e90d864bc9783 P c0742b25a62b4793bf20353f0d1f2a95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0742b25a62b4793bf20353f0d1f2a95" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 35609 } } peers { permanent_uuid: "49fa8338699749498ff4283df5548ab3" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 36055 } } peers { permanent_uuid: "7a79b23b92bc49bda09c1b6e71137e49" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 46139 } }
I20260812 09:45:50.919986 26551 raft_consensus.cc:374] T 2e737064c8e34096976e90d864bc9783 P c0742b25a62b4793bf20353f0d1f2a95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:45:50.920017 26551 raft_consensus.cc:729] T 2e737064c8e34096976e90d864bc9783 P c0742b25a62b4793bf20353f0d1f2a95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0742b25a62b4793bf20353f0d1f2a95, State: Initialized, Role: FOLLOWER
I20260812 09:45:50.920127 26551 consensus_queue.cc:260] T 2e737064c8e34096976e90d864bc9783 P c0742b25a62b4793bf20353f0d1f2a95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0742b25a62b4793bf20353f0d1f2a95" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 35609 } } peers { permanent_uuid: "49fa8338699749498ff4283df5548ab3" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 36055 } } peers { permanent_uuid: "7a79b23b92bc49bda09c1b6e71137e49" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 46139 } }
I20260812 09:45:50.920361 26551 ts_tablet_manager.cc:1428] T 2e737064c8e34096976e90d864bc9783 P c0742b25a62b4793bf20353f0d1f2a95: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 09:45:50.921072 26552 tablet_bootstrap.cc:492] T 2e737064c8e34096976e90d864bc9783 P 7a79b23b92bc49bda09c1b6e71137e49: Bootstrap starting.
I20260812 09:45:50.921703 26552 tablet_bootstrap.cc:654] T 2e737064c8e34096976e90d864bc9783 P 7a79b23b92bc49bda09c1b6e71137e49: Neither blocks nor log segments found. Creating new log.
I20260812 09:45:50.922816 26552 tablet_bootstrap.cc:492] T 2e737064c8e34096976e90d864bc9783 P 7a79b23b92bc49bda09c1b6e71137e49: No bootstrap required, opened a new log
I20260812 09:45:50.922895 26552 ts_tablet_manager.cc:1397] T 2e737064c8e34096976e90d864bc9783 P 7a79b23b92bc49bda09c1b6e71137e49: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260812 09:45:50.923048 26550 tablet_bootstrap.cc:492] T 2e737064c8e34096976e90d864bc9783 P 49fa8338699749498ff4283df5548ab3: No bootstrap required, opened a new log
I20260812 09:45:50.923141 26550 ts_tablet_manager.cc:1397] T 2e737064c8e34096976e90d864bc9783 P 49fa8338699749498ff4283df5548ab3: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260812 09:45:50.923300 26552 raft_consensus.cc:348] T 2e737064c8e34096976e90d864bc9783 P 7a79b23b92bc49bda09c1b6e71137e49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0742b25a62b4793bf20353f0d1f2a95" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 35609 } } peers { permanent_uuid: "49fa8338699749498ff4283df5548ab3" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 36055 } } peers { permanent_uuid: "7a79b23b92bc49bda09c1b6e71137e49" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 46139 } }
I20260812 09:45:50.923401 26552 raft_consensus.cc:374] T 2e737064c8e34096976e90d864bc9783 P 7a79b23b92bc49bda09c1b6e71137e49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:45:50.923441 26552 raft_consensus.cc:729] T 2e737064c8e34096976e90d864bc9783 P 7a79b23b92bc49bda09c1b6e71137e49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a79b23b92bc49bda09c1b6e71137e49, State: Initialized, Role: FOLLOWER
I20260812 09:45:50.923555 26552 consensus_queue.cc:260] T 2e737064c8e34096976e90d864bc9783 P 7a79b23b92bc49bda09c1b6e71137e49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0742b25a62b4793bf20353f0d1f2a95" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 35609 } } peers { permanent_uuid: "49fa8338699749498ff4283df5548ab3" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 36055 } } peers { permanent_uuid: "7a79b23b92bc49bda09c1b6e71137e49" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 46139 } }
I20260812 09:45:50.923511 26550 raft_consensus.cc:348] T 2e737064c8e34096976e90d864bc9783 P 49fa8338699749498ff4283df5548ab3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0742b25a62b4793bf20353f0d1f2a95" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 35609 } } peers { permanent_uuid: "49fa8338699749498ff4283df5548ab3" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 36055 } } peers { permanent_uuid: "7a79b23b92bc49bda09c1b6e71137e49" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 46139 } }
I20260812 09:45:50.923679 26550 raft_consensus.cc:374] T 2e737064c8e34096976e90d864bc9783 P 49fa8338699749498ff4283df5548ab3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:45:50.923703 26550 raft_consensus.cc:729] T 2e737064c8e34096976e90d864bc9783 P 49fa8338699749498ff4283df5548ab3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49fa8338699749498ff4283df5548ab3, State: Initialized, Role: FOLLOWER
I20260812 09:45:50.923806 26552 ts_tablet_manager.cc:1428] T 2e737064c8e34096976e90d864bc9783 P 7a79b23b92bc49bda09c1b6e71137e49: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260812 09:45:50.923832 26550 consensus_queue.cc:260] T 2e737064c8e34096976e90d864bc9783 P 49fa8338699749498ff4283df5548ab3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0742b25a62b4793bf20353f0d1f2a95" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 35609 } } peers { permanent_uuid: "49fa8338699749498ff4283df5548ab3" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 36055 } } peers { permanent_uuid: "7a79b23b92bc49bda09c1b6e71137e49" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 46139 } }
I20260812 09:45:50.924108 26550 ts_tablet_manager.cc:1428] T 2e737064c8e34096976e90d864bc9783 P 49fa8338699749498ff4283df5548ab3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 09:45:51.110304 26568 raft_consensus.cc:482] T 2e737064c8e34096976e90d864bc9783 P c0742b25a62b4793bf20353f0d1f2a95 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 09:45:51.110476 26568 raft_consensus.cc:504] T 2e737064c8e34096976e90d864bc9783 P c0742b25a62b4793bf20353f0d1f2a95 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0742b25a62b4793bf20353f0d1f2a95" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 35609 } } peers { permanent_uuid: "49fa8338699749498ff4283df5548ab3" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 36055 } } peers { permanent_uuid: "7a79b23b92bc49bda09c1b6e71137e49" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 46139 } }
I20260812 09:45:51.110916 26568 leader_election.cc:288] T 2e737064c8e34096976e90d864bc9783 P c0742b25a62b4793bf20353f0d1f2a95 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49fa8338699749498ff4283df5548ab3 (127.13.99.2:36055), 7a79b23b92bc49bda09c1b6e71137e49 (127.13.99.3:46139)
I20260812 09:45:51.111402 26180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e737064c8e34096976e90d864bc9783" candidate_uuid: "c0742b25a62b4793bf20353f0d1f2a95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49fa8338699749498ff4283df5548ab3" is_pre_election: true
I20260812 09:45:51.111454 26424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e737064c8e34096976e90d864bc9783" candidate_uuid: "c0742b25a62b4793bf20353f0d1f2a95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a79b23b92bc49bda09c1b6e71137e49" is_pre_election: true
I20260812 09:45:51.111567 26180 raft_consensus.cc:2454] T 2e737064c8e34096976e90d864bc9783 P 49fa8338699749498ff4283df5548ab3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0742b25a62b4793bf20353f0d1f2a95 in term 0.
I20260812 09:45:51.111594 26424 raft_consensus.cc:2454] T 2e737064c8e34096976e90d864bc9783 P 7a79b23b92bc49bda09c1b6e71137e49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0742b25a62b4793bf20353f0d1f2a95 in term 0.
I20260812 09:45:51.112052 25863 leader_election.cc:302] T 2e737064c8e34096976e90d864bc9783 P c0742b25a62b4793bf20353f0d1f2a95 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49fa8338699749498ff4283df5548ab3, c0742b25a62b4793bf20353f0d1f2a95; no voters:
I20260812 09:45:51.112260 26568 raft_consensus.cc:2781] T 2e737064c8e34096976e90d864bc9783 P c0742b25a62b4793bf20353f0d1f2a95 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 09:45:51.112331 26568 raft_consensus.cc:482] T 2e737064c8e34096976e90d864bc9783 P c0742b25a62b4793bf20353f0d1f2a95 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 09:45:51.112385 26568 raft_consensus.cc:3037] T 2e737064c8e34096976e90d864bc9783 P c0742b25a62b4793bf20353f0d1f2a95 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:45:51.113538 26568 raft_consensus.cc:504] T 2e737064c8e34096976e90d864bc9783 P c0742b25a62b4793bf20353f0d1f2a95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0742b25a62b4793bf20353f0d1f2a95" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 35609 } } peers { permanent_uuid: "49fa8338699749498ff4283df5548ab3" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 36055 } } peers { permanent_uuid: "7a79b23b92bc49bda09c1b6e71137e49" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 46139 } }
I20260812 09:45:51.113879 26568 leader_election.cc:288] T 2e737064c8e34096976e90d864bc9783 P c0742b25a62b4793bf20353f0d1f2a95 [CANDIDATE]: Term 1 election: Requested vote from peers 49fa8338699749498ff4283df5548ab3 (127.13.99.2:36055), 7a79b23b92bc49bda09c1b6e71137e49 (127.13.99.3:46139)
I20260812 09:45:51.114306 26180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e737064c8e34096976e90d864bc9783" candidate_uuid: "c0742b25a62b4793bf20353f0d1f2a95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49fa8338699749498ff4283df5548ab3"
I20260812 09:45:51.114306 26424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e737064c8e34096976e90d864bc9783" candidate_uuid: "c0742b25a62b4793bf20353f0d1f2a95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a79b23b92bc49bda09c1b6e71137e49"
I20260812 09:45:51.114437 26424 raft_consensus.cc:3037] T 2e737064c8e34096976e90d864bc9783 P 7a79b23b92bc49bda09c1b6e71137e49 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:45:51.114439 26180 raft_consensus.cc:3037] T 2e737064c8e34096976e90d864bc9783 P 49fa8338699749498ff4283df5548ab3 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:45:51.115603 26180 raft_consensus.cc:2454] T 2e737064c8e34096976e90d864bc9783 P 49fa8338699749498ff4283df5548ab3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0742b25a62b4793bf20353f0d1f2a95 in term 1.
I20260812 09:45:51.115603 26424 raft_consensus.cc:2454] T 2e737064c8e34096976e90d864bc9783 P 7a79b23b92bc49bda09c1b6e71137e49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0742b25a62b4793bf20353f0d1f2a95 in term 1.
I20260812 09:45:51.115880 25863 leader_election.cc:302] T 2e737064c8e34096976e90d864bc9783 P c0742b25a62b4793bf20353f0d1f2a95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a79b23b92bc49bda09c1b6e71137e49, c0742b25a62b4793bf20353f0d1f2a95; no voters:
I20260812 09:45:51.116091 26568 raft_consensus.cc:2781] T 2e737064c8e34096976e90d864bc9783 P c0742b25a62b4793bf20353f0d1f2a95 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 09:45:51.116256 26568 raft_consensus.cc:686] T 2e737064c8e34096976e90d864bc9783 P c0742b25a62b4793bf20353f0d1f2a95 [term 1 LEADER]: Becoming Leader. State: Replica: c0742b25a62b4793bf20353f0d1f2a95, State: Running, Role: LEADER
I20260812 09:45:51.116420 26568 consensus_queue.cc:237] T 2e737064c8e34096976e90d864bc9783 P c0742b25a62b4793bf20353f0d1f2a95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0742b25a62b4793bf20353f0d1f2a95" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 35609 } } peers { permanent_uuid: "49fa8338699749498ff4283df5548ab3" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 36055 } } peers { permanent_uuid: "7a79b23b92bc49bda09c1b6e71137e49" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 46139 } }
I20260812 09:45:51.118125 25720 catalog_manager.cc:5495] T 2e737064c8e34096976e90d864bc9783 P c0742b25a62b4793bf20353f0d1f2a95 reported cstate change: term changed from 0 to 1, leader changed from <none> to c0742b25a62b4793bf20353f0d1f2a95 (127.13.99.1). New cstate: current_term: 1 leader_uuid: "c0742b25a62b4793bf20353f0d1f2a95" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0742b25a62b4793bf20353f0d1f2a95" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 35609 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49fa8338699749498ff4283df5548ab3" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 36055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a79b23b92bc49bda09c1b6e71137e49" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 46139 } health_report { overall_health: UNKNOWN } } }
I20260812 09:45:51.166421 26692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:51.163240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48086 (local address 127.13.99.1:35609)
0812 09:45:51.163491 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:51.163507 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:51.163519 (+ 12us) server_negotiation.cc:407] Connection header received
0812 09:45:51.163559 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:51.163560 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:51.163599 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:51.163676 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 09:45:51.163958 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:51.164540 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:51.165383 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:51.165658 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:51.165736 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:51.165983 (+ 247us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 09:45:51.166055 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 09:45:51.166108 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:51.166205 (+ 97us) server_negotiation.cc:299] Negotiation successful
0812 09:45:51.166248 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":100,"threads_started":1}
I20260812 09:45:51.176698 26700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:51.173245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.3:46139 (local address 127.13.99.1:33695)
0812 09:45:51.173531 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:51.173544 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:51.173676 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:51.173952 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:51.173955 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:51.173962 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 09:45:51.174049 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:51.174054 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:51.174857 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:51.174859 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:51.175551 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:51.175561 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:51.176050 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:51.176053 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:51.176114 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:51.176160 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 09:45:51.176435 (+ 275us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 09:45:51.176440 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 09:45:51.176516 (+ 76us) client_negotiation.cc:240] Negotiation successful
0812 09:45:51.176583 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":104,"threads_started":1}
I20260812 09:45:51.176693 26543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:51.173339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.1:33695 (local address 127.13.99.3:46139)
0812 09:45:51.173492 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:51.173495 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:51.173642 (+ 147us) server_negotiation.cc:407] Connection header received
0812 09:45:51.173769 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:51.173771 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:51.173817 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:51.173920 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 09:45:51.174154 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:51.174757 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:51.175692 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:51.175941 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:51.176031 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:51.176283 (+ 252us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 09:45:51.176352 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 09:45:51.176410 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:51.176537 (+ 127us) server_negotiation.cc:299] Negotiation successful
0812 09:45:51.176592 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 09:45:51.183348 26180 raft_consensus.cc:1261] T 2e737064c8e34096976e90d864bc9783 P 49fa8338699749498ff4283df5548ab3 [term 1 FOLLOWER]: Refusing update from remote peer c0742b25a62b4793bf20353f0d1f2a95: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 09:45:51.183348 26424 raft_consensus.cc:1261] T 2e737064c8e34096976e90d864bc9783 P 7a79b23b92bc49bda09c1b6e71137e49 [term 1 FOLLOWER]: Refusing update from remote peer c0742b25a62b4793bf20353f0d1f2a95: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 09:45:51.183755 26686 consensus_queue.cc:1035] T 2e737064c8e34096976e90d864bc9783 P c0742b25a62b4793bf20353f0d1f2a95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49fa8338699749498ff4283df5548ab3" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 36055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 09:45:51.184010 26568 consensus_queue.cc:1035] T 2e737064c8e34096976e90d864bc9783 P c0742b25a62b4793bf20353f0d1f2a95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a79b23b92bc49bda09c1b6e71137e49" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 46139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 09:45:51.199762 26692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:51.196295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.3:42819 (local address 127.13.99.1:35609)
0812 09:45:51.196396 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:51.196400 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:51.196643 (+ 243us) server_negotiation.cc:407] Connection header received
0812 09:45:51.196758 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:51.196761 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:51.196806 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:51.196890 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 09:45:51.197157 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:51.197765 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:51.198723 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:51.198961 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:51.199076 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:51.199334 (+ 258us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 09:45:51.199411 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 09:45:51.199493 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:51.199584 (+ 91us) server_negotiation.cc:299] Negotiation successful
0812 09:45:51.199636 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260812 09:45:51.199802 26707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:51.196229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.1:35609 (local address 127.13.99.3:42819)
0812 09:45:51.196548 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:51.196563 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:51.196671 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:51.196913 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:51.196917 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:51.196925 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 09:45:51.197040 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:51.197047 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:51.197876 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:51.197881 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:51.198597 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:51.198608 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:51.199077 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:51.199081 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:51.199174 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:51.199210 (+ 36us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 09:45:51.199498 (+ 288us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 09:45:51.199508 (+ 10us) client_negotiation.cc:769] Sending connection context
0812 09:45:51.199576 (+ 68us) client_negotiation.cc:240] Negotiation successful
0812 09:45:51.199668 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":71,"threads_started":1}
I20260812 09:45:51.220216 25724 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55160:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 09:45:51.220362 25724 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55160:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 09:45:51.222277 25724 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7fb7f7ee6bd04ed7bb41ec750ae6d72a: Sending DeleteTablet for 3 replicas of tablet 2e737064c8e34096976e90d864bc9783
I20260812 09:45:51.222849 26394 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e737064c8e34096976e90d864bc9783 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 09:45:51 UTC) from {username='oryx', principal='oryx/127.13.99.62@KRBTEST.COM'} at 127.0.0.1:41666
I20260812 09:45:51.222867 26140 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e737064c8e34096976e90d864bc9783 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 09:45:51 UTC) from {username='oryx', principal='oryx/127.13.99.62@KRBTEST.COM'} at 127.0.0.1:54622
I20260812 09:45:51.223029 26593 tablet_replica.cc:330] T 2e737064c8e34096976e90d864bc9783 P 7a79b23b92bc49bda09c1b6e71137e49: stopping tablet replica
I20260812 09:45:51.223028 26594 tablet_replica.cc:330] T 2e737064c8e34096976e90d864bc9783 P 49fa8338699749498ff4283df5548ab3: stopping tablet replica
I20260812 09:45:51.222993 25936 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e737064c8e34096976e90d864bc9783 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 09:45:51 UTC) from {username='oryx', principal='oryx/127.13.99.62@KRBTEST.COM'} at 127.0.0.1:48072
I20260812 09:45:51.223131 26593 raft_consensus.cc:2229] T 2e737064c8e34096976e90d864bc9783 P 7a79b23b92bc49bda09c1b6e71137e49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 09:45:51.223130 26594 raft_consensus.cc:2229] T 2e737064c8e34096976e90d864bc9783 P 49fa8338699749498ff4283df5548ab3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 09:45:51.223230 26593 raft_consensus.cc:2258] T 2e737064c8e34096976e90d864bc9783 P 7a79b23b92bc49bda09c1b6e71137e49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 09:45:51.223234 26594 raft_consensus.cc:2258] T 2e737064c8e34096976e90d864bc9783 P 49fa8338699749498ff4283df5548ab3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 09:45:51.223413 26592 tablet_replica.cc:330] T 2e737064c8e34096976e90d864bc9783 P c0742b25a62b4793bf20353f0d1f2a95: stopping tablet replica
I20260812 09:45:51.223551 26592 raft_consensus.cc:2229] T 2e737064c8e34096976e90d864bc9783 P c0742b25a62b4793bf20353f0d1f2a95 [term 1 LEADER]: Raft consensus shutting down.
I20260812 09:45:51.223734 26592 raft_consensus.cc:2258] T 2e737064c8e34096976e90d864bc9783 P c0742b25a62b4793bf20353f0d1f2a95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 09:45:51.223968 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 25840
I20260812 09:45:51.224195 26594 ts_tablet_manager.cc:1905] T 2e737064c8e34096976e90d864bc9783 P 49fa8338699749498ff4283df5548ab3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 09:45:51.227010 26594 ts_tablet_manager.cc:1918] T 2e737064c8e34096976e90d864bc9783 P 49fa8338699749498ff4283df5548ab3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 09:45:51.227082 26594 log.cc:1196] T 2e737064c8e34096976e90d864bc9783 P 49fa8338699749498ff4283df5548ab3: Deleting WAL directory at /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/minicluster-data/ts-1/wal/wals/2e737064c8e34096976e90d864bc9783
I20260812 09:45:51.227319 26594 ts_tablet_manager.cc:1939] T 2e737064c8e34096976e90d864bc9783 P 49fa8338699749498ff4283df5548ab3: Deleting consensus metadata
I20260812 09:45:51.232679 25701 catalog_manager.cc:4841] TS 49fa8338699749498ff4283df5548ab3 (127.13.99.2:36055): tablet 2e737064c8e34096976e90d864bc9783 (table test-table [id=d7905afda122460392dd1ef0bbd29b06]) successfully deleted
I20260812 09:45:51.235231 26593 ts_tablet_manager.cc:1905] T 2e737064c8e34096976e90d864bc9783 P 7a79b23b92bc49bda09c1b6e71137e49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 09:45:51.244935 26593 ts_tablet_manager.cc:1918] T 2e737064c8e34096976e90d864bc9783 P 7a79b23b92bc49bda09c1b6e71137e49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 09:45:51.245030 26593 log.cc:1196] T 2e737064c8e34096976e90d864bc9783 P 7a79b23b92bc49bda09c1b6e71137e49: Deleting WAL directory at /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786527905902985-13708-0/minicluster-data/ts-2/wal/wals/2e737064c8e34096976e90d864bc9783
I20260812 09:45:51.245303 26593 ts_tablet_manager.cc:1939] T 2e737064c8e34096976e90d864bc9783 P 7a79b23b92bc49bda09c1b6e71137e49: Deleting consensus metadata
I20260812 09:45:51.246387 25701 catalog_manager.cc:4841] TS 7a79b23b92bc49bda09c1b6e71137e49 (127.13.99.3:46139): tablet 2e737064c8e34096976e90d864bc9783 (table test-table [id=d7905afda122460392dd1ef0bbd29b06]) successfully deleted
W20260812 09:45:51.247225 25702 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.99.1:35609 (error 108)
I20260812 09:45:51.247561 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 26044
I20260812 09:45:51.252213 26725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:51.251713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.1:35609 (local address 127.0.0.1:48100)
0812 09:45:51.251981 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:51.252073 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.1:35609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":56,"threads_started":1}
W20260812 09:45:51.252395 25702 catalog_manager.cc:4584] TS c0742b25a62b4793bf20353f0d1f2a95 (127.13.99.1:35609): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2e737064c8e34096976e90d864bc9783: Network error: Client connection negotiation failed: client connection to 127.13.99.1:35609: connect: Connection refused (error 111)
I20260812 09:45:51.262305 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 26262
I20260812 09:45:51.281234 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 25677
2026-08-12T09:45:51Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3677 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786527905902985-13708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 09:45:51 dist-test-slave-jq4t krb5kdc[26729](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 09:45:51 dist-test-slave-jq4t krb5kdc[26729](info): set up 2 sockets
Aug 12 09:45:51 dist-test-slave-jq4t krb5kdc[26729](info): commencing operation
krb5kdc: starting...
W20260812 09:45:52.586419 13708 mini_kdc.cc:121] Time spent starting KDC: real 1.270s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786527905902985-13708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786527905902985-13708-0/krb5kdc/test-user.keytab.
Aug 12 09:45:52 dist-test-slave-jq4t krb5kdc[26729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527952, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T09:45:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T09:45:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.99.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.99.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.99.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab.
Entry for principal kudu/127.13.99.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab.
I20260812 09:45:52.713961 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786527905902985-13708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786527905902985-13708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786527905902985-13708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.62:37295
--webserver_interface=127.13.99.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab
--principal=kudu/127.13.99.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:38857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.99.62:37295
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:45:52.829258 26748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:45:52.829495 26748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:45:52.829537 26748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:45:52.833432 26748 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 09:45:52.833487 26748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:45:52.833505 26748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:45:52.833521 26748 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 09:45:52.833536 26748 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 09:45:52.838548 26748 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:38857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786527905902985-13708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786527905902985-13708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.99.62:37295
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.62:37295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.99.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26748
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786527905902985-13708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:45:52.839480 26748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:45:52.840440 26748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:45:52.849246 26758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:52.849315 26759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:52.849339 26764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:45:52.849308 26748 server_base.cc:1034] running on GCE node
I20260812 09:45:52.850190 26748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:45:52.851348 26748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:45:52.852551 26748 hybrid_clock.cc:648] HybridClock initialized: now 1786527952852538 us; error 30 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-12T09:45:52Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1570 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 09:45:52 dist-test-slave-jq4t krb5kdc[26772](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 09:45:52 dist-test-slave-jq4t krb5kdc[26772](info): set up 2 sockets
Aug 12 09:45:52 dist-test-slave-jq4t krb5kdc[26772](info): commencing operation
krb5kdc: starting...
W20260812 09:45:54.152033 13708 mini_kdc.cc:121] Time spent starting KDC: real 1.265s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/krb5kdc/test-user.keytab.
Aug 12 09:45:54 dist-test-slave-jq4t krb5kdc[26772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527954, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T09:45:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T09:45:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.99.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.99.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.99.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab.
Entry for principal kudu/127.13.99.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab.
I20260812 09:45:54.297652 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.62:40203
--webserver_interface=127.13.99.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab
--principal=kudu/127.13.99.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:34809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.99.62:40203
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:45:54.411914 26794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:45:54.412197 26794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:45:54.412238 26794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:45:54.416374 26794 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 09:45:54.416430 26794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:45:54.416455 26794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:45:54.416517 26794 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 09:45:54.416553 26794 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 09:45:54.421810 26794 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:34809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.99.62:40203
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab
--principal=kudu/127.13.99.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.62:40203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.99.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26794
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:45:54.422858 26794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:45:54.423895 26794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:45:54.432456 26806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:54.432461 26809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:45:54.432659 26794 server_base.cc:1034] running on GCE node
W20260812 09:45:54.432461 26803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:45:54.433327 26794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:45:54.434442 26794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:45:54.435642 26794 hybrid_clock.cc:648] HybridClock initialized: now 1786527954435630 us; error 26 us; skew 500 ppm
Aug 12 09:45:54 dist-test-slave-jq4t krb5kdc[26772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527954, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:45:54.442288 26794 init.cc:377] Logged in from keytab as kudu/127.13.99.62@KRBTEST.COM (short username kudu)
I20260812 09:45:54.443279 26794 webserver.cc:460] Webserver started at http://127.13.99.62:32871/ using document root <none> and password file <none>
I20260812 09:45:54.443872 26794 fs_manager.cc:359] Metadata directory not provided
I20260812 09:45:54.443979 26794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:45:54.444221 26794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:45:54.446384 26794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/master-0/data/instance:
uuid: "e18b37b67e9e43a4ae2071eb0da45cfb"
format_stamp: "Formatted at 2026-08-12 09:45:54 on dist-test-slave-jq4t"
server_key: "26549b7f7cac8c8316c35c4fa15227f3"
server_key_iv: "dc0f30604c0d24eb998226587d199bae"
server_key_version: "encryptionkey@0"
I20260812 09:45:54.446983 26794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/master-0/wal/instance:
uuid: "e18b37b67e9e43a4ae2071eb0da45cfb"
format_stamp: "Formatted at 2026-08-12 09:45:54 on dist-test-slave-jq4t"
server_key: "26549b7f7cac8c8316c35c4fa15227f3"
server_key_iv: "dc0f30604c0d24eb998226587d199bae"
server_key_version: "encryptionkey@0"
I20260812 09:45:54.450909 26794 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260812 09:45:54.453547 26819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:54.454836 26794 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 09:45:54.455003 26794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/master-0/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/master-0/wal
uuid: "e18b37b67e9e43a4ae2071eb0da45cfb"
format_stamp: "Formatted at 2026-08-12 09:45:54 on dist-test-slave-jq4t"
server_key: "26549b7f7cac8c8316c35c4fa15227f3"
server_key_iv: "dc0f30604c0d24eb998226587d199bae"
server_key_version: "encryptionkey@0"
I20260812 09:45:54.455123 26794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:45:54.472962 26794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:45:54.476310 26794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:45:54.476621 26794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:45:54.483999 26794 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.62:40203
I20260812 09:45:54.484019 26892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.62:40203 every 8 connection(s)
I20260812 09:45:54.485003 26794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
I20260812 09:45:54.487769 26894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:45:54.493526 26894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e18b37b67e9e43a4ae2071eb0da45cfb: Bootstrap starting.
I20260812 09:45:54.494740 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 26794
I20260812 09:45:54.494894 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/master-0/wal/instance
I20260812 09:45:54.495201 13708 external_mini_cluster.cc:1428] Setting key 0c7eb1555686a6a93ce976658b780dd9
I20260812 09:45:54.496591 26894 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e18b37b67e9e43a4ae2071eb0da45cfb: Neither blocks nor log segments found. Creating new log.
I20260812 09:45:54.497411 26894 log.cc:824] T 00000000000000000000000000000000 P e18b37b67e9e43a4ae2071eb0da45cfb: Log is configured to *not* fsync() on all Append() calls
I20260812 09:45:54.499516 26894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e18b37b67e9e43a4ae2071eb0da45cfb: No bootstrap required, opened a new log
I20260812 09:45:54.503329 26894 raft_consensus.cc:348] T 00000000000000000000000000000000 P e18b37b67e9e43a4ae2071eb0da45cfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e18b37b67e9e43a4ae2071eb0da45cfb" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 40203 } }
I20260812 09:45:54.503602 26894 raft_consensus.cc:374] T 00000000000000000000000000000000 P e18b37b67e9e43a4ae2071eb0da45cfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:45:54.503696 26894 raft_consensus.cc:729] T 00000000000000000000000000000000 P e18b37b67e9e43a4ae2071eb0da45cfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e18b37b67e9e43a4ae2071eb0da45cfb, State: Initialized, Role: FOLLOWER
Aug 12 09:45:54 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527954, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
I20260812 09:45:54.504199 26894 consensus_queue.cc:260] T 00000000000000000000000000000000 P e18b37b67e9e43a4ae2071eb0da45cfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e18b37b67e9e43a4ae2071eb0da45cfb" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 40203 } }
I20260812 09:45:54.504395 26894 raft_consensus.cc:388] T 00000000000000000000000000000000 P e18b37b67e9e43a4ae2071eb0da45cfb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 09:45:54.504511 26894 raft_consensus.cc:482] T 00000000000000000000000000000000 P e18b37b67e9e43a4ae2071eb0da45cfb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 09:45:54.504622 26894 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e18b37b67e9e43a4ae2071eb0da45cfb [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:45:54.505801 26894 raft_consensus.cc:504] T 00000000000000000000000000000000 P e18b37b67e9e43a4ae2071eb0da45cfb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e18b37b67e9e43a4ae2071eb0da45cfb" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 40203 } }
I20260812 09:45:54.506202 26894 leader_election.cc:302] T 00000000000000000000000000000000 P e18b37b67e9e43a4ae2071eb0da45cfb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e18b37b67e9e43a4ae2071eb0da45cfb; no voters:
I20260812 09:45:54.506570 26894 leader_election.cc:288] T 00000000000000000000000000000000 P e18b37b67e9e43a4ae2071eb0da45cfb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 09:45:54.506649 26903 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e18b37b67e9e43a4ae2071eb0da45cfb [term 1 FOLLOWER]: Leader election won for term 1
I20260812 09:45:54.506866 26903 raft_consensus.cc:686] T 00000000000000000000000000000000 P e18b37b67e9e43a4ae2071eb0da45cfb [term 1 LEADER]: Becoming Leader. State: Replica: e18b37b67e9e43a4ae2071eb0da45cfb, State: Running, Role: LEADER
I20260812 09:45:54.507341 26903 consensus_queue.cc:237] T 00000000000000000000000000000000 P e18b37b67e9e43a4ae2071eb0da45cfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e18b37b67e9e43a4ae2071eb0da45cfb" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 40203 } }
I20260812 09:45:54.508095 26894 sys_catalog.cc:564] T 00000000000000000000000000000000 P e18b37b67e9e43a4ae2071eb0da45cfb [sys.catalog]: configured and running, proceeding with master startup.
I20260812 09:45:54.508674 26905 sys_catalog.cc:455] T 00000000000000000000000000000000 P e18b37b67e9e43a4ae2071eb0da45cfb [sys.catalog]: SysCatalogTable state changed. Reason: New leader e18b37b67e9e43a4ae2071eb0da45cfb. Latest consensus state: current_term: 1 leader_uuid: "e18b37b67e9e43a4ae2071eb0da45cfb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e18b37b67e9e43a4ae2071eb0da45cfb" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 40203 } } }
I20260812 09:45:54.508793 26905 sys_catalog.cc:458] T 00000000000000000000000000000000 P e18b37b67e9e43a4ae2071eb0da45cfb [sys.catalog]: This master's current role is: LEADER
I20260812 09:45:54.509406 26904 sys_catalog.cc:455] T 00000000000000000000000000000000 P e18b37b67e9e43a4ae2071eb0da45cfb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e18b37b67e9e43a4ae2071eb0da45cfb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e18b37b67e9e43a4ae2071eb0da45cfb" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 40203 } } }
I20260812 09:45:54.509529 26904 sys_catalog.cc:458] T 00000000000000000000000000000000 P e18b37b67e9e43a4ae2071eb0da45cfb [sys.catalog]: This master's current role is: LEADER
I20260812 09:45:54.510742 26897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:54.496442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35258 (local address 127.13.99.62:40203)
0812 09:45:54.496938 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:54.496957 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:54.496994 (+ 37us) server_negotiation.cc:407] Connection header received
0812 09:45:54.497725 (+ 731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:54.497745 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:54.498189 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:54.498582 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:54.499434 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:54.500553 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:54.501459 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:54.501969 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:54.502153 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:54.505009 (+ 2856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:54.505014 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:54.505029 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:54.505064 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:54.506997 (+ 1933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:54.507585 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:54.507589 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:54.507594 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:54.507704 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:54.508022 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:54.508026 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:54.508027 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:54.508615 (+ 588us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:54.508799 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:54.509284 (+ 485us) server_negotiation.cc:299] Negotiation successful
0812 09:45:54.509632 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":148,"threads_started":1}
W20260812 09:45:54.514679 26927 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e18b37b67e9e43a4ae2071eb0da45cfb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 09:45:54.514787 26927 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 09:45:54.514901 26908 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 09:45:54.515686 26908 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 09:45:54.522343 26908 catalog_manager.cc:1347] Generated new cluster ID: 647777eb95e44e468cd74f02b61ae207
I20260812 09:45:54.522437 26908 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 09:45:54.534919 26908 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 09:45:54.535966 26908 catalog_manager.cc:1504] Loading token signing keys...
I20260812 09:45:54.542589 26908 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e18b37b67e9e43a4ae2071eb0da45cfb: Generated new TSK 0
I20260812 09:45:54.543582 26908 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.99.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.99.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.99.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.1.keytab.
Entry for principal kudu/127.13.99.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.1.keytab.
I20260812 09:45:54.581884 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.99.1:0
--local_ip_for_outbound_sockets=127.13.99.1
--webserver_interface=127.13.99.1
--webserver_port=0
--tserver_master_addrs=127.13.99.62:40203
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.1.keytab
--principal=kudu/127.13.99.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:34809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:45:54.699532 26932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:45:54.699788 26932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:45:54.699832 26932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:45:54.704301 26932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:45:54.704382 26932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:45:54.704453 26932 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.99.1
I20260812 09:45:54.710435 26932 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:34809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.1.keytab
--principal=kudu/127.13.99.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.99.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.99.1
--webserver_port=0
--tserver_master_addrs=127.13.99.62:40203
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26932
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.99.1
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:45:54.711537 26932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:45:54.712544 26932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:45:54.722332 26948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:54.722303 26945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:54.722285 26946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:45:54.722597 26932 server_base.cc:1034] running on GCE node
I20260812 09:45:54.723061 26932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:45:54.723753 26932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:45:54.724921 26932 hybrid_clock.cc:648] HybridClock initialized: now 1786527954724894 us; error 37 us; skew 500 ppm
Aug 12 09:45:54 dist-test-slave-jq4t krb5kdc[26772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527954, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:45:54.727782 26932 init.cc:377] Logged in from keytab as kudu/127.13.99.1@KRBTEST.COM (short username kudu)
I20260812 09:45:54.728739 26932 webserver.cc:460] Webserver started at http://127.13.99.1:41023/ using document root <none> and password file <none>
I20260812 09:45:54.729568 26932 fs_manager.cc:359] Metadata directory not provided
I20260812 09:45:54.729686 26932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:45:54.729975 26932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:45:54.732599 26932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/ts-0/data/instance:
uuid: "32a49ce87cd04b8a92df85441589fd3b"
format_stamp: "Formatted at 2026-08-12 09:45:54 on dist-test-slave-jq4t"
server_key: "952b5280d28d732836f3f38d8fafee63"
server_key_iv: "5a06699cb077f7ab5bbe82eaefb724c7"
server_key_version: "encryptionkey@0"
I20260812 09:45:54.733184 26932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/ts-0/wal/instance:
uuid: "32a49ce87cd04b8a92df85441589fd3b"
format_stamp: "Formatted at 2026-08-12 09:45:54 on dist-test-slave-jq4t"
server_key: "952b5280d28d732836f3f38d8fafee63"
server_key_iv: "5a06699cb077f7ab5bbe82eaefb724c7"
server_key_version: "encryptionkey@0"
I20260812 09:45:54.736884 26932 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 09:45:54.739284 26960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:54.740535 26932 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 09:45:54.740686 26932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/ts-0/wal
uuid: "32a49ce87cd04b8a92df85441589fd3b"
format_stamp: "Formatted at 2026-08-12 09:45:54 on dist-test-slave-jq4t"
server_key: "952b5280d28d732836f3f38d8fafee63"
server_key_iv: "5a06699cb077f7ab5bbe82eaefb724c7"
server_key_version: "encryptionkey@0"
I20260812 09:45:54.740793 26932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:45:54.755307 26932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:45:54.758558 26932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:45:54.758786 26932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:45:54.759490 26932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 09:45:54.760363 26932 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:45:54.760437 26932 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:54.760520 26932 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:45:54.760556 26932 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:54.769577 26932 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.1:39839
I20260812 09:45:54.769649 27142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.1:39839 every 8 connection(s)
I20260812 09:45:54.770787 26932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/ts-0/data/info.pb
I20260812 09:45:54.779263 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 26932
I20260812 09:45:54.779417 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/ts-0/wal/instance
I20260812 09:45:54.779737 13708 external_mini_cluster.cc:1428] Setting key bf0178aaf8a759021cd9d9a7a585c449
Aug 12 09:45:54 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527954, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.1@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
I20260812 09:45:54.784966 26897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:54.772761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.1:44341 (local address 127.13.99.62:40203)
0812 09:45:54.772907 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:54.772911 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:54.774057 (+ 1146us) server_negotiation.cc:407] Connection header received
0812 09:45:54.775007 (+ 950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:54.775011 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:54.775071 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:54.775162 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:54.776357 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:54.777008 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:54.778195 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:54.778443 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:54.778560 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:54.781384 (+ 2824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:54.781387 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:54.781389 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:54.781413 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:54.782827 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:54.783354 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:54.783358 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:54.783360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:54.783448 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:54.783795 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:54.783800 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:54.783802 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:54.783994 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:54.784104 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:54.784653 (+ 549us) server_negotiation.cc:299] Negotiation successful
0812 09:45:54.784782 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260812 09:45:54.785758 27146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:54.773175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:40203 (local address 127.13.99.1:44341)
0812 09:45:54.773890 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:54.773940 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:54.774796 (+ 856us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:54.775354 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:54.775361 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:54.775728 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:45:54.776207 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:54.776221 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:54.777142 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:54.777145 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:54.778057 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:54.778067 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:54.778601 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:54.778606 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:54.778773 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:54.779436 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:45:54.779463 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:45:54.781220 (+ 1757us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:45:54.782953 (+ 1733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:54.782959 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:54.782976 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:54.783243 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:54.783545 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:54.783548 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:54.783550 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:54.783697 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:54.784104 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:45:54.784112 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:45:54.784365 (+ 253us) client_negotiation.cc:769] Sending connection context
0812 09:45:54.784602 (+ 237us) client_negotiation.cc:240] Negotiation successful
0812 09:45:54.784929 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":124,"threads_started":1}
I20260812 09:45:54.787163 27143 heartbeater.cc:347] Connected to a master server at 127.13.99.62:40203
I20260812 09:45:54.787439 27143 heartbeater.cc:464] Registering TS with master...
I20260812 09:45:54.788106 27143 heartbeater.cc:511] Master 127.13.99.62:40203 requested a full tablet report, sending...
I20260812 09:45:54.789821 26841 ts_manager.cc:194] Registered new tserver with Master: 32a49ce87cd04b8a92df85441589fd3b (127.13.99.1:39839)
WARNING: no policy specified for kudu/127.13.99.2@KRBTEST.COM; defaulting to no policy
I20260812 09:45:54.791033 26841 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.99.1@KRBTEST.COM'} at 127.13.99.1:44341
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.99.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.99.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.2.keytab.
Entry for principal kudu/127.13.99.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.2.keytab.
I20260812 09:45:54.811703 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.99.2:0
--local_ip_for_outbound_sockets=127.13.99.2
--webserver_interface=127.13.99.2
--webserver_port=0
--tserver_master_addrs=127.13.99.62:40203
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.2.keytab
--principal=kudu/127.13.99.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:34809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:45:54.928642 27149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:45:54.928915 27149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:45:54.929013 27149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:45:54.933029 27149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:45:54.933111 27149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:45:54.933203 27149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.99.2
I20260812 09:45:54.938323 27149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:34809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.2.keytab
--principal=kudu/127.13.99.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.99.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.99.2
--webserver_port=0
--tserver_master_addrs=127.13.99.62:40203
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.99.2
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:45:54.939426 27149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:45:54.940569 27149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:45:54.950580 27159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:54.950580 27160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:54.950613 27165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:45:54.950896 27149 server_base.cc:1034] running on GCE node
I20260812 09:45:54.951296 27149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:45:54.951915 27149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:45:54.953097 27149 hybrid_clock.cc:648] HybridClock initialized: now 1786527954953059 us; error 44 us; skew 500 ppm
Aug 12 09:45:54 dist-test-slave-jq4t krb5kdc[26772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527954, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:45:54.955968 27149 init.cc:377] Logged in from keytab as kudu/127.13.99.2@KRBTEST.COM (short username kudu)
I20260812 09:45:54.957065 27149 webserver.cc:460] Webserver started at http://127.13.99.2:35363/ using document root <none> and password file <none>
I20260812 09:45:54.957640 27149 fs_manager.cc:359] Metadata directory not provided
I20260812 09:45:54.957705 27149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:45:54.957949 27149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:45:54.960157 27149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/ts-1/data/instance:
uuid: "be42e30e7ee748c88710e6b198bfd294"
format_stamp: "Formatted at 2026-08-12 09:45:54 on dist-test-slave-jq4t"
server_key: "4194b2f761549aaefcd8bc532afca0a3"
server_key_iv: "e8f93d41bf561781e7e22a365dd0093d"
server_key_version: "encryptionkey@0"
I20260812 09:45:54.960752 27149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/ts-1/wal/instance:
uuid: "be42e30e7ee748c88710e6b198bfd294"
format_stamp: "Formatted at 2026-08-12 09:45:54 on dist-test-slave-jq4t"
server_key: "4194b2f761549aaefcd8bc532afca0a3"
server_key_iv: "e8f93d41bf561781e7e22a365dd0093d"
server_key_version: "encryptionkey@0"
I20260812 09:45:54.964468 27149 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260812 09:45:54.967015 27181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:54.968200 27149 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 09:45:54.968325 27149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/ts-1/wal
uuid: "be42e30e7ee748c88710e6b198bfd294"
format_stamp: "Formatted at 2026-08-12 09:45:54 on dist-test-slave-jq4t"
server_key: "4194b2f761549aaefcd8bc532afca0a3"
server_key_iv: "e8f93d41bf561781e7e22a365dd0093d"
server_key_version: "encryptionkey@0"
I20260812 09:45:54.968430 27149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:45:54.987382 27149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:45:54.991137 27149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:45:54.991384 27149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:45:54.992069 27149 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 09:45:54.993017 27149 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:45:54.993094 27149 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:54.993175 27149 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:45:54.993219 27149 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:55.002646 27149 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.2:45953
I20260812 09:45:55.002669 27364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.2:45953 every 8 connection(s)
I20260812 09:45:55.003599 27149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/ts-1/data/info.pb
I20260812 09:45:55.009171 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 27149
I20260812 09:45:55.009311 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/ts-1/wal/instance
I20260812 09:45:55.009654 13708 external_mini_cluster.cc:1428] Setting key 6bbe98dd4b7eb084d6f2967900d68a89
Aug 12 09:45:55 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527954, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.2@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
I20260812 09:45:55.019300 26897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:55.005643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.2:46225 (local address 127.13.99.62:40203)
0812 09:45:55.005786 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:55.005789 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:55.006972 (+ 1183us) server_negotiation.cc:407] Connection header received
0812 09:45:55.007996 (+ 1024us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:55.008001 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:55.008067 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:55.008161 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:55.009828 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:55.010540 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:55.012013 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:55.012403 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:55.012559 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:55.015428 (+ 2869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:55.015437 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:55.015440 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:55.015465 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:55.016995 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:55.017558 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:55.017564 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:55.017567 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:55.017672 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:55.018024 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:55.018031 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:55.018035 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:55.018319 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:55.018443 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:55.018954 (+ 511us) server_negotiation.cc:299] Negotiation successful
0812 09:45:55.019101 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260812 09:45:55.020051 27370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:55.006036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:40203 (local address 127.13.99.2:46225)
0812 09:45:55.006790 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:55.006835 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:55.007738 (+ 903us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:55.008345 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:55.008353 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:55.008831 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:45:55.009589 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:55.009616 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:55.010681 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:55.010685 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:55.011864 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:55.011875 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:55.012597 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:55.012600 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:55.012741 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:55.013393 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:45:55.013414 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:45:55.015242 (+ 1828us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:45:55.017161 (+ 1919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:55.017170 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:55.017194 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:55.017451 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:55.017775 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:55.017777 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:55.017779 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:55.017907 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:55.018447 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:45:55.018452 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:45:55.018693 (+ 241us) client_negotiation.cc:769] Sending connection context
0812 09:45:55.018852 (+ 159us) client_negotiation.cc:240] Negotiation successful
0812 09:45:55.019128 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":112,"threads_started":1}
I20260812 09:45:55.021462 27367 heartbeater.cc:347] Connected to a master server at 127.13.99.62:40203
WARNING: no policy specified for kudu/127.13.99.3@KRBTEST.COM; defaulting to no policy
I20260812 09:45:55.021770 27367 heartbeater.cc:464] Registering TS with master...
I20260812 09:45:55.022282 27367 heartbeater.cc:511] Master 127.13.99.62:40203 requested a full tablet report, sending...
I20260812 09:45:55.023427 26841 ts_manager.cc:194] Registered new tserver with Master: be42e30e7ee748c88710e6b198bfd294 (127.13.99.2:45953)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.99.3@KRBTEST.COM" created.
I20260812 09:45:55.024223 26841 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.99.2@KRBTEST.COM'} at 127.13.99.2:46225
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.99.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.3.keytab.
Entry for principal kudu/127.13.99.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.3.keytab.
I20260812 09:45:55.039805 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.99.3:0
--local_ip_for_outbound_sockets=127.13.99.3
--webserver_interface=127.13.99.3
--webserver_port=0
--tserver_master_addrs=127.13.99.62:40203
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.3.keytab
--principal=kudu/127.13.99.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:34809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:45:55.155071 27378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:45:55.155344 27378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:45:55.155416 27378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:45:55.159632 27378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:45:55.159709 27378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:45:55.159795 27378 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.99.3
I20260812 09:45:55.164997 27378 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:34809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.3.keytab
--principal=kudu/127.13.99.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.99.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.99.3
--webserver_port=0
--tserver_master_addrs=127.13.99.62:40203
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27378
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.99.3
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:45:55.166013 27378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:45:55.167027 27378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:45:55.176685 27389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:55.176695 27387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:45:55.176761 27378 server_base.cc:1034] running on GCE node
W20260812 09:45:55.176693 27391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:45:55.177496 27378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:45:55.178112 27378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:45:55.179275 27378 hybrid_clock.cc:648] HybridClock initialized: now 1786527955179264 us; error 35 us; skew 500 ppm
Aug 12 09:45:55 dist-test-slave-jq4t krb5kdc[26772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527955, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:45:55.182155 27378 init.cc:377] Logged in from keytab as kudu/127.13.99.3@KRBTEST.COM (short username kudu)
I20260812 09:45:55.183035 27378 webserver.cc:460] Webserver started at http://127.13.99.3:43109/ using document root <none> and password file <none>
I20260812 09:45:55.183605 27378 fs_manager.cc:359] Metadata directory not provided
I20260812 09:45:55.183667 27378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:45:55.183917 27378 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:45:55.185971 27378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/ts-2/data/instance:
uuid: "ad994f351acd46dc962078e805fdb720"
format_stamp: "Formatted at 2026-08-12 09:45:55 on dist-test-slave-jq4t"
server_key: "9b304d1489019d04717729fc64d8d4a3"
server_key_iv: "bd88e84abe9b3b4f215a85d46231add7"
server_key_version: "encryptionkey@0"
I20260812 09:45:55.186545 27378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/ts-2/wal/instance:
uuid: "ad994f351acd46dc962078e805fdb720"
format_stamp: "Formatted at 2026-08-12 09:45:55 on dist-test-slave-jq4t"
server_key: "9b304d1489019d04717729fc64d8d4a3"
server_key_iv: "bd88e84abe9b3b4f215a85d46231add7"
server_key_version: "encryptionkey@0"
I20260812 09:45:55.190181 27378 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260812 09:45:55.192674 27404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:55.193960 27378 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 09:45:55.194095 27378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/ts-2/wal
uuid: "ad994f351acd46dc962078e805fdb720"
format_stamp: "Formatted at 2026-08-12 09:45:55 on dist-test-slave-jq4t"
server_key: "9b304d1489019d04717729fc64d8d4a3"
server_key_iv: "bd88e84abe9b3b4f215a85d46231add7"
server_key_version: "encryptionkey@0"
I20260812 09:45:55.194202 27378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:45:55.216387 27378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:45:55.219746 27378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:45:55.219916 27378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:45:55.220507 27378 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 09:45:55.221395 27378 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:45:55.221443 27378 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:55.221483 27378 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:45:55.221498 27378 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:55.230738 27378 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.3:35825
I20260812 09:45:55.230769 27616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.3:35825 every 8 connection(s)
I20260812 09:45:55.231657 27378 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/ts-2/data/info.pb
I20260812 09:45:55.236864 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 27378
I20260812 09:45:55.237015 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/ts-2/wal/instance
I20260812 09:45:55.237329 13708 external_mini_cluster.cc:1428] Setting key b11a673ea32bb72e5b5d03d64ef2fe89
Aug 12 09:45:55 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527955, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.3@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
I20260812 09:45:55.245741 26897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:55.233583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.3:55565 (local address 127.13.99.62:40203)
0812 09:45:55.233731 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:55.233735 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:55.234878 (+ 1143us) server_negotiation.cc:407] Connection header received
0812 09:45:55.235832 (+ 954us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:55.235836 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:55.235896 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:55.236007 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:55.237518 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:55.238092 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:55.239557 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:55.239812 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:55.239897 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:55.242333 (+ 2436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:55.242336 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:55.242338 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:55.242361 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:55.243737 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:55.244304 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:55.244307 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:55.244308 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:55.244388 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:55.244790 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:55.244792 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:55.244793 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:55.244974 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:55.245054 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:55.245521 (+ 467us) server_negotiation.cc:299] Negotiation successful
0812 09:45:55.245623 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260812 09:45:55.246501 27620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:55.233983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:40203 (local address 127.13.99.3:55565)
0812 09:45:55.234712 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:55.234753 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:55.235617 (+ 864us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:55.236191 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:55.236198 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:55.236598 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:45:55.237308 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:55.237329 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:55.238242 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:55.238246 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:55.239437 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:55.239449 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:55.239963 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:55.239966 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:55.240108 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:55.240744 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:45:55.240761 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:45:55.242185 (+ 1424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:45:55.243879 (+ 1694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:55.243884 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:55.243895 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:55.244164 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:55.244517 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:45:55.244524 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:45:55.244525 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:45:55.244696 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:45:55.245066 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:45:55.245071 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:45:55.245290 (+ 219us) client_negotiation.cc:769] Sending connection context
0812 09:45:55.245442 (+ 152us) client_negotiation.cc:240] Negotiation successful
0812 09:45:55.245686 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":126,"threads_started":1}
I20260812 09:45:55.247712 27617 heartbeater.cc:347] Connected to a master server at 127.13.99.62:40203
I20260812 09:45:55.247967 27617 heartbeater.cc:464] Registering TS with master...
I20260812 09:45:55.248471 27617 heartbeater.cc:511] Master 127.13.99.62:40203 requested a full tablet report, sending...
I20260812 09:45:55.249663 26841 ts_manager.cc:194] Registered new tserver with Master: ad994f351acd46dc962078e805fdb720 (127.13.99.3:35825)
I20260812 09:45:55.250309 26841 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.99.3@KRBTEST.COM'} at 127.13.99.3:55565
I20260812 09:45:55.251540 13708 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.13.99.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.99.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.99.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/krb5kdc/oryx_127.13.99.1.keytab.
Entry for principal oryx/127.13.99.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/krb5kdc/oryx_127.13.99.1.keytab.
WARNING: no policy specified for oryx/127.13.99.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.99.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.99.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/krb5kdc/oryx_127.13.99.2.keytab.
Entry for principal oryx/127.13.99.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/krb5kdc/oryx_127.13.99.2.keytab.
WARNING: no policy specified for oryx/127.13.99.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.99.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.99.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/krb5kdc/oryx_127.13.99.3.keytab.
Entry for principal oryx/127.13.99.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/krb5kdc/oryx_127.13.99.3.keytab.
I20260812 09:45:55.327059 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 26932
I20260812 09:45:55.342365 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 27149
I20260812 09:45:55.352384 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 27378
I20260812 09:45:55.361898 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 26794
I20260812 09:45:55.373482 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.62:40203
--webserver_interface=127.13.99.62
--webserver_port=32871
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab
--principal=kudu/127.13.99.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:34809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.99.62:40203
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:45:55.490569 27645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:45:55.490828 27645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:45:55.490908 27645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:45:55.494917 27645 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 09:45:55.494972 27645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:45:55.494992 27645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:45:55.495038 27645 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 09:45:55.495072 27645 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 09:45:55.500206 27645 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:34809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.99.62:40203
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab
--principal=kudu/127.13.99.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.62:40203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.99.62
--webserver_port=32871
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27645
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:45:55.501243 27645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:45:55.502254 27645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:45:55.511256 27656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:55.511272 27652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:55.511273 27654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:45:55.511718 27645 server_base.cc:1034] running on GCE node
I20260812 09:45:55.512318 27645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:45:55.513547 27645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:45:55.514711 27645 hybrid_clock.cc:648] HybridClock initialized: now 1786527955514694 us; error 31 us; skew 500 ppm
Aug 12 09:45:55 dist-test-slave-jq4t krb5kdc[26772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527955, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:45:55.521106 27645 init.cc:377] Logged in from keytab as kudu/127.13.99.62@KRBTEST.COM (short username kudu)
I20260812 09:45:55.522115 27645 webserver.cc:460] Webserver started at http://127.13.99.62:32871/ using document root <none> and password file <none>
I20260812 09:45:55.522723 27645 fs_manager.cc:359] Metadata directory not provided
I20260812 09:45:55.522845 27645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:45:55.526935 27645 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 09:45:55.529188 27668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:55.530458 27645 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 09:45:55.530613 27645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/master-0/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/master-0/wal
uuid: "e18b37b67e9e43a4ae2071eb0da45cfb"
format_stamp: "Formatted at 2026-08-12 09:45:54 on dist-test-slave-jq4t"
server_key: "26549b7f7cac8c8316c35c4fa15227f3"
server_key_iv: "dc0f30604c0d24eb998226587d199bae"
server_key_version: "encryptionkey@0"
I20260812 09:45:55.531185 27645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:45:55.562173 27645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:45:55.565668 27645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:45:55.565888 27645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:45:55.573477 27645 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.62:40203
I20260812 09:45:55.573768 27766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.62:40203 every 8 connection(s)
I20260812 09:45:55.574599 27645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
I20260812 09:45:55.579071 27772 sys_catalog.cc:263] Verifying existing consensus state
I20260812 09:45:55.580813 27772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e18b37b67e9e43a4ae2071eb0da45cfb: Bootstrap starting.
I20260812 09:45:55.580956 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 27645
I20260812 09:45:55.581481 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.99.1:39839
--local_ip_for_outbound_sockets=127.13.99.1
--tserver_master_addrs=127.13.99.62:40203
--webserver_port=41023
--webserver_interface=127.13.99.1
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.1.keytab
--principal=kudu/127.13.99.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:34809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.13.99.1
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/krb5kdc/oryx_127.13.99.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 09:45:55.599166 27772 log.cc:824] T 00000000000000000000000000000000 P e18b37b67e9e43a4ae2071eb0da45cfb: Log is configured to *not* fsync() on all Append() calls
I20260812 09:45:55.608572 27772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e18b37b67e9e43a4ae2071eb0da45cfb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 09:45:55.609027 27772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e18b37b67e9e43a4ae2071eb0da45cfb: Bootstrap complete.
I20260812 09:45:55.615254 27772 raft_consensus.cc:348] T 00000000000000000000000000000000 P e18b37b67e9e43a4ae2071eb0da45cfb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e18b37b67e9e43a4ae2071eb0da45cfb" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 40203 } }
I20260812 09:45:55.615867 27772 raft_consensus.cc:729] T 00000000000000000000000000000000 P e18b37b67e9e43a4ae2071eb0da45cfb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e18b37b67e9e43a4ae2071eb0da45cfb, State: Initialized, Role: FOLLOWER
I20260812 09:45:55.616611 27772 consensus_queue.cc:260] T 00000000000000000000000000000000 P e18b37b67e9e43a4ae2071eb0da45cfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e18b37b67e9e43a4ae2071eb0da45cfb" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 40203 } }
I20260812 09:45:55.616827 27772 raft_consensus.cc:388] T 00000000000000000000000000000000 P e18b37b67e9e43a4ae2071eb0da45cfb [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 09:45:55.616932 27772 raft_consensus.cc:482] T 00000000000000000000000000000000 P e18b37b67e9e43a4ae2071eb0da45cfb [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 09:45:55.617043 27772 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e18b37b67e9e43a4ae2071eb0da45cfb [term 1 FOLLOWER]: Advancing to term 2
I20260812 09:45:55.619818 27772 raft_consensus.cc:504] T 00000000000000000000000000000000 P e18b37b67e9e43a4ae2071eb0da45cfb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e18b37b67e9e43a4ae2071eb0da45cfb" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 40203 } }
I20260812 09:45:55.620314 27772 leader_election.cc:302] T 00000000000000000000000000000000 P e18b37b67e9e43a4ae2071eb0da45cfb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e18b37b67e9e43a4ae2071eb0da45cfb; no voters:
I20260812 09:45:55.621965 27772 leader_election.cc:288] T 00000000000000000000000000000000 P e18b37b67e9e43a4ae2071eb0da45cfb [CANDIDATE]: Term 2 election: Requested vote from peers
I20260812 09:45:55.622053 27783 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e18b37b67e9e43a4ae2071eb0da45cfb [term 2 FOLLOWER]: Leader election won for term 2
I20260812 09:45:55.624004 27783 raft_consensus.cc:686] T 00000000000000000000000000000000 P e18b37b67e9e43a4ae2071eb0da45cfb [term 2 LEADER]: Becoming Leader. State: Replica: e18b37b67e9e43a4ae2071eb0da45cfb, State: Running, Role: LEADER
I20260812 09:45:55.624333 27783 consensus_queue.cc:237] T 00000000000000000000000000000000 P e18b37b67e9e43a4ae2071eb0da45cfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e18b37b67e9e43a4ae2071eb0da45cfb" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 40203 } }
I20260812 09:45:55.625238 27772 sys_catalog.cc:564] T 00000000000000000000000000000000 P e18b37b67e9e43a4ae2071eb0da45cfb [sys.catalog]: configured and running, proceeding with master startup.
I20260812 09:45:55.625759 27785 sys_catalog.cc:455] T 00000000000000000000000000000000 P e18b37b67e9e43a4ae2071eb0da45cfb [sys.catalog]: SysCatalogTable state changed. Reason: New leader e18b37b67e9e43a4ae2071eb0da45cfb. Latest consensus state: current_term: 2 leader_uuid: "e18b37b67e9e43a4ae2071eb0da45cfb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e18b37b67e9e43a4ae2071eb0da45cfb" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 40203 } } }
I20260812 09:45:55.625978 27785 sys_catalog.cc:458] T 00000000000000000000000000000000 P e18b37b67e9e43a4ae2071eb0da45cfb [sys.catalog]: This master's current role is: LEADER
I20260812 09:45:55.626344 27784 sys_catalog.cc:455] T 00000000000000000000000000000000 P e18b37b67e9e43a4ae2071eb0da45cfb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e18b37b67e9e43a4ae2071eb0da45cfb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e18b37b67e9e43a4ae2071eb0da45cfb" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 40203 } } }
I20260812 09:45:55.626502 27784 sys_catalog.cc:458] T 00000000000000000000000000000000 P e18b37b67e9e43a4ae2071eb0da45cfb [sys.catalog]: This master's current role is: LEADER
I20260812 09:45:55.626528 27788 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 09:45:55.630245 27788 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 09:45:55.631011 27788 catalog_manager.cc:1259] Loaded cluster ID: 647777eb95e44e468cd74f02b61ae207
I20260812 09:45:55.631856 27788 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 09:45:55.633757 27788 catalog_manager.cc:1504] Loading token signing keys...
I20260812 09:45:55.634559 27788 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e18b37b67e9e43a4ae2071eb0da45cfb: Loaded TSK: 0
I20260812 09:45:55.636718 27788 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 09:45:55.734216 27776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:45:55.734480 27776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:45:55.734547 27776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:45:55.738678 27776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:45:55.738736 27776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:45:55.738829 27776 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.99.1
I20260812 09:45:55.744038 27776 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:34809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/krb5kdc/oryx_127.13.99.1.keytab
--principal=oryx/127.13.99.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.99.1:39839
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.99.1
--webserver_port=41023
--tserver_master_addrs=127.13.99.62:40203
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27776
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.99.1
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:45:55.745169 27776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:45:55.746227 27776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:45:55.756939 27823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:55.757638 27829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:55.758059 27826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:45:55.758788 27776 server_base.cc:1034] running on GCE node
I20260812 09:45:55.759224 27776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:45:55.759903 27776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:45:55.761096 27776 hybrid_clock.cc:648] HybridClock initialized: now 1786527955761074 us; error 38 us; skew 500 ppm
Aug 12 09:45:55 dist-test-slave-jq4t krb5kdc[26772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527955, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.99.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:45:55.764096 27776 init.cc:377] Logged in from keytab as oryx/127.13.99.1@KRBTEST.COM (short username oryx)
I20260812 09:45:55.765096 27776 webserver.cc:460] Webserver started at http://127.13.99.1:41023/ using document root <none> and password file <none>
I20260812 09:45:55.765683 27776 fs_manager.cc:359] Metadata directory not provided
I20260812 09:45:55.765796 27776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:45:55.769873 27776 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 09:45:55.772130 27836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:55.773481 27776 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 09:45:55.773619 27776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/ts-0/wal
uuid: "32a49ce87cd04b8a92df85441589fd3b"
format_stamp: "Formatted at 2026-08-12 09:45:54 on dist-test-slave-jq4t"
server_key: "952b5280d28d732836f3f38d8fafee63"
server_key_iv: "5a06699cb077f7ab5bbe82eaefb724c7"
server_key_version: "encryptionkey@0"
I20260812 09:45:55.774187 27776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:45:55.798420 27776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:45:55.802256 27776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:45:55.802479 27776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:45:55.803184 27776 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 09:45:55.804060 27776 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:45:55.804134 27776 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:55.804199 27776 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:45:55.804250 27776 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:55.814198 27776 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.1:39839
I20260812 09:45:55.814224 28030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.1:39839 every 8 connection(s)
I20260812 09:45:55.815198 27776 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/ts-0/data/info.pb
I20260812 09:45:55.820320 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 27776
I20260812 09:45:55.820870 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.99.2:45953
--local_ip_for_outbound_sockets=127.13.99.2
--tserver_master_addrs=127.13.99.62:40203
--webserver_port=35363
--webserver_interface=127.13.99.2
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.2.keytab
--principal=kudu/127.13.99.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:34809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.13.99.2
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/krb5kdc/oryx_127.13.99.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 09:45:55 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.1@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 09:45:55 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.1@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 09:45:55.830673 28036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:55.817389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:40203 (local address 127.13.99.1:41367)
0812 09:45:55.818399 (+ 1010us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:55.818443 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:55.819291 (+ 848us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:55.820428 (+ 1137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:55.820438 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:55.820834 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:45:55.821381 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:55.821397 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:55.823179 (+ 1782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:55.823184 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:55.824705 (+ 1521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:55.824720 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:55.825507 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:55.825514 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:55.825768 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:55.826514 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:45:55.826532 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:45:55.829629 (+ 3097us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.99.62:40203: Server oryx/127.13.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":814,"thread_start_us":149,"threads_started":1}
W20260812 09:45:55.830664 28035 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 09:45:55.817238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.1:41367 (local address 127.13.99.62:40203)
0812 09:45:55.817738 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:55.817764 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:55.818565 (+ 801us) server_negotiation.cc:407] Connection header received
0812 09:45:55.819667 (+ 1102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:55.819678 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:55.819964 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:55.820348 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:55.821565 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:55.823032 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:55.824843 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:55.825323 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:55.825450 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:55.829880 (+ 4430us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.99.1:41367: BlockingRecv error: recv got EOF from 127.13.99.1:41367 (error 108)
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":121,"threads_started":1}
W20260812 09:45:55.831187 28031 heartbeater.cc:644] Failed to heartbeat to 127.13.99.62:40203 (0 consecutive failures): Not authorized: Failed to ping master at 127.13.99.62:40203: Client connection negotiation failed: client connection to 127.13.99.62:40203: Server oryx/127.13.99.62@KRBTEST.COM not found in Kerberos database
Aug 12 09:45:55 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.1@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 09:45:55 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.1@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 09:45:55.837217 28036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:55.831645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:40203 (local address 127.13.99.1:53557)
0812 09:45:55.831852 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:55.831870 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:55.831979 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:55.832265 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:55.832268 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:55.832444 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:45:55.832606 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:55.832612 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:55.833358 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:55.833363 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:55.833930 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:55.833938 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:55.834455 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:55.834458 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:55.834552 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:55.835070 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:45:55.835084 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:45:55.837077 (+ 1993us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.99.62:40203: Server oryx/127.13.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104}
W20260812 09:45:55.837311 28035 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 09:45:55.831774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.1:53557 (local address 127.13.99.62:40203)
0812 09:45:55.831924 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:55.831928 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:55.831945 (+ 17us) server_negotiation.cc:407] Connection header received
0812 09:45:55.832101 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:55.832104 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:55.832158 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:55.832248 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:55.832718 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:55.833241 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:55.834055 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:55.834335 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:55.834450 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:55.837197 (+ 2747us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.99.1:53557: BlockingRecv error: recv got EOF from 127.13.99.1:53557 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 12 09:45:55 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.1@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 09:45:55 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.1@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 09:45:55.842805 28036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:55.837668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:40203 (local address 127.13.99.1:44207)
0812 09:45:55.837800 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:55.837815 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:55.837895 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:55.838205 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:55.838208 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:55.838368 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:45:55.838459 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:55.838464 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:55.839117 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:55.839120 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:55.840000 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:55.840012 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:55.840412 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:55.840415 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:55.840526 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:55.840937 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:45:55.840950 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:45:55.842689 (+ 1739us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.99.62:40203: Server oryx/127.13.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":53}
W20260812 09:45:55.842916 28035 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 09:45:55.837893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.1:44207 (local address 127.13.99.62:40203)
0812 09:45:55.838007 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:55.838013 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:55.838029 (+ 16us) server_negotiation.cc:407] Connection header received
0812 09:45:55.838059 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:55.838061 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:55.838094 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:55.838173 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:55.838567 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:55.839020 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:55.840117 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:55.840311 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:55.840391 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:55.842802 (+ 2411us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.99.1:44207: BlockingRecv error: recv got EOF from 127.13.99.1:44207 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260812 09:45:55.843032 28031 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 09:45:55.939877 28037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:45:55.940125 28037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:45:55.940166 28037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:45:55.944198 28037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:45:55.944257 28037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:45:55.944315 28037 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.99.2
I20260812 09:45:55.949559 28037 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:34809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/krb5kdc/oryx_127.13.99.2.keytab
--principal=oryx/127.13.99.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.99.2:45953
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.99.2
--webserver_port=35363
--tserver_master_addrs=127.13.99.62:40203
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28037
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.99.2
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:45:55.950613 28037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:45:55.951675 28037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:45:55.961333 28049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:55.961359 28047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:55.961366 28045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:45:55.961604 28037 server_base.cc:1034] running on GCE node
I20260812 09:45:55.962112 28037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:45:55.962734 28037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:45:55.963901 28037 hybrid_clock.cc:648] HybridClock initialized: now 1786527955963879 us; error 37 us; skew 500 ppm
Aug 12 09:45:55 dist-test-slave-jq4t krb5kdc[26772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527955, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.99.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:45:55.966769 28037 init.cc:377] Logged in from keytab as oryx/127.13.99.2@KRBTEST.COM (short username oryx)
I20260812 09:45:55.967671 28037 webserver.cc:460] Webserver started at http://127.13.99.2:35363/ using document root <none> and password file <none>
I20260812 09:45:55.968241 28037 fs_manager.cc:359] Metadata directory not provided
I20260812 09:45:55.968359 28037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:45:55.972265 28037 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260812 09:45:55.974519 28062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:55.975742 28037 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 09:45:55.975903 28037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/ts-1/wal
uuid: "be42e30e7ee748c88710e6b198bfd294"
format_stamp: "Formatted at 2026-08-12 09:45:54 on dist-test-slave-jq4t"
server_key: "4194b2f761549aaefcd8bc532afca0a3"
server_key_iv: "e8f93d41bf561781e7e22a365dd0093d"
server_key_version: "encryptionkey@0"
I20260812 09:45:55.976523 28037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:45:55.997906 28037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:45:56.001405 28037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:45:56.001626 28037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:45:56.002264 28037 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 09:45:56.003197 28037 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:45:56.003274 28037 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:56.003351 28037 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:45:56.003402 28037 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:56.012755 28037 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.2:45953
I20260812 09:45:56.012790 28260 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.2:45953 every 8 connection(s)
I20260812 09:45:56.013813 28037 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/ts-1/data/info.pb
I20260812 09:45:56.019441 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 28037
I20260812 09:45:56.020067 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.99.3:35825
--local_ip_for_outbound_sockets=127.13.99.3
--tserver_master_addrs=127.13.99.62:40203
--webserver_port=43109
--webserver_interface=127.13.99.3
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.3.keytab
--principal=kudu/127.13.99.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:34809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.13.99.3
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/krb5kdc/oryx_127.13.99.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 09:45:56 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.2@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 09:45:56 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.2@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
W20260812 09:45:56.027228 28035 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 09:45:56.015662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.2:51671 (local address 127.13.99.62:40203)
0812 09:45:56.015806 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:56.015810 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:56.016850 (+ 1040us) server_negotiation.cc:407] Connection header received
0812 09:45:56.017781 (+ 931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:56.017785 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:56.017855 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:56.017967 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:56.020420 (+ 2453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:56.021041 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:56.022625 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:56.023017 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:56.023151 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:56.027101 (+ 3950us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.99.2:51671: BlockingRecv error: recv got EOF from 127.13.99.2:51671 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260812 09:45:56.027880 28265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:56.015990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:40203 (local address 127.13.99.2:51671)
0812 09:45:56.016681 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:56.016724 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:56.017564 (+ 840us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:56.018167 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:56.018174 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:56.018539 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:45:56.020180 (+ 1641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:56.020208 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:56.021191 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:56.021198 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:56.022482 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:56.022493 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:56.023178 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:56.023181 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:56.023321 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:56.023937 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:45:56.023963 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:45:56.026959 (+ 2996us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.99.62:40203: Server oryx/127.13.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":114,"threads_started":1}
W20260812 09:45:56.028445 28261 heartbeater.cc:644] Failed to heartbeat to 127.13.99.62:40203 (0 consecutive failures): Not authorized: Failed to ping master at 127.13.99.62:40203: Client connection negotiation failed: client connection to 127.13.99.62:40203: Server oryx/127.13.99.62@KRBTEST.COM not found in Kerberos database
Aug 12 09:45:56 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.2@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 09:45:56 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.2@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 09:45:56.034890 28265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:56.028916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:40203 (local address 127.13.99.2:53149)
0812 09:45:56.029111 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:56.029132 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:56.029196 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:56.029494 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:56.029496 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:56.029626 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:45:56.029716 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:56.029720 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:56.030872 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:56.030876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:56.031426 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:56.031435 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:56.032092 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:56.032095 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:56.032184 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:56.032774 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:45:56.032786 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:45:56.034792 (+ 2006us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.99.62:40203: Server oryx/127.13.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125}
W20260812 09:45:56.035066 28035 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 09:45:56.029040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.2:53149 (local address 127.13.99.62:40203)
0812 09:45:56.029201 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:56.029207 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:56.029253 (+ 46us) server_negotiation.cc:407] Connection header received
0812 09:45:56.029308 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:56.029312 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:56.029382 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:56.029472 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:56.029833 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:56.030697 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:56.031556 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:56.031963 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:56.032092 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:56.034954 (+ 2862us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.99.2:53149: BlockingRecv error: recv got EOF from 127.13.99.2:53149 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 12 09:45:56 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.2@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 09:45:56 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.2@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 09:45:56.040872 28265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:56.035622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:40203 (local address 127.13.99.2:52873)
0812 09:45:56.035765 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:56.035780 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:56.035855 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:56.036118 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:56.036121 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:56.036278 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:45:56.036379 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:56.036385 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:56.037121 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:56.037124 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:56.037974 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:56.037984 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:56.038402 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:56.038405 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:56.038487 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:56.038922 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:45:56.038936 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:45:56.040755 (+ 1819us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.99.62:40203: Server oryx/127.13.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":63}
W20260812 09:45:56.040952 28035 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 09:45:56.035798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.2:52873 (local address 127.13.99.62:40203)
0812 09:45:56.035935 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:56.035939 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:56.035951 (+ 12us) server_negotiation.cc:407] Connection header received
0812 09:45:56.035982 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:56.035984 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:56.036017 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:56.036082 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:56.036525 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:56.037018 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:56.038092 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:56.038293 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:56.038374 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:56.040872 (+ 2498us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.99.2:52873: BlockingRecv error: recv got EOF from 127.13.99.2:52873 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260812 09:45:56.041378 28261 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 09:45:56.138684 28266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:45:56.138985 28266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:45:56.139079 28266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:45:56.143067 28266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:45:56.143150 28266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:45:56.143239 28266 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.99.3
I20260812 09:45:56.148463 28266 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:34809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/krb5kdc/oryx_127.13.99.3.keytab
--principal=oryx/127.13.99.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.99.3:35825
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.99.3
--webserver_port=43109
--tserver_master_addrs=127.13.99.62:40203
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28266
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.99.3
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:45:56.149616 28266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:45:56.150684 28266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:45:56.160705 28278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:56.160698 28276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:45:56.160698 28273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:45:56.160898 28266 server_base.cc:1034] running on GCE node
I20260812 09:45:56.161371 28266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:45:56.162011 28266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:45:56.163206 28266 hybrid_clock.cc:648] HybridClock initialized: now 1786527956163168 us; error 47 us; skew 500 ppm
Aug 12 09:45:56 dist-test-slave-jq4t krb5kdc[26772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527956, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.99.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:45:56.166167 28266 init.cc:377] Logged in from keytab as oryx/127.13.99.3@KRBTEST.COM (short username oryx)
I20260812 09:45:56.167001 28266 webserver.cc:460] Webserver started at http://127.13.99.3:43109/ using document root <none> and password file <none>
I20260812 09:45:56.167625 28266 fs_manager.cc:359] Metadata directory not provided
I20260812 09:45:56.167704 28266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:45:56.171617 28266 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260812 09:45:56.173796 28288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:56.175184 28266 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 09:45:56.175330 28266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/ts-2/wal
uuid: "ad994f351acd46dc962078e805fdb720"
format_stamp: "Formatted at 2026-08-12 09:45:55 on dist-test-slave-jq4t"
server_key: "9b304d1489019d04717729fc64d8d4a3"
server_key_iv: "bd88e84abe9b3b4f215a85d46231add7"
server_key_version: "encryptionkey@0"
I20260812 09:45:56.175887 28266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:45:56.196233 28266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:45:56.199663 28266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:45:56.199877 28266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:45:56.200551 28266 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 09:45:56.201409 28266 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:45:56.201467 28266 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:56.201506 28266 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:45:56.201522 28266 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 09:45:56.210832 28266 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.3:35825
I20260812 09:45:56.210920 28493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.3:35825 every 8 connection(s)
I20260812 09:45:56.211869 28266 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786527905902985-13708-0/minicluster-data/ts-2/data/info.pb
I20260812 09:45:56.218559 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 28266
Aug 12 09:45:56 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.3@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 09:45:56 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.3@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
W20260812 09:45:56.223388 28035 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 09:45:56.213794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.3:51899 (local address 127.13.99.62:40203)
0812 09:45:56.213931 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:56.213934 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:56.214926 (+ 992us) server_negotiation.cc:407] Connection header received
0812 09:45:56.215833 (+ 907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:56.215836 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:56.215899 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:56.216009 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:56.217280 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:56.217813 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:56.219037 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:56.219258 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:56.219522 (+ 264us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:56.223291 (+ 3769us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.99.3:51899: BlockingRecv error: recv got EOF from 127.13.99.3:51899 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260812 09:45:56.224061 28497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:56.214098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:40203 (local address 127.13.99.3:51899)
0812 09:45:56.214755 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:56.214796 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:56.215627 (+ 831us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:56.216204 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:56.216212 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:56.216639 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:45:56.217127 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:56.217141 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:56.217957 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:56.217965 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:56.218896 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:56.218906 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:56.219462 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:56.219469 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:56.219750 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:56.220455 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:45:56.220470 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:45:56.223170 (+ 2700us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.99.62:40203: Server oryx/127.13.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":118,"threads_started":1}
W20260812 09:45:56.224733 28495 heartbeater.cc:644] Failed to heartbeat to 127.13.99.62:40203 (0 consecutive failures): Not authorized: Failed to ping master at 127.13.99.62:40203: Client connection negotiation failed: client connection to 127.13.99.62:40203: Server oryx/127.13.99.62@KRBTEST.COM not found in Kerberos database
I20260812 09:45:56.229178 28500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:56.219603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33282 (local address 127.13.99.62:40203)
0812 09:45:56.219860 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:56.219865 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:56.219884 (+ 19us) server_negotiation.cc:407] Connection header received
0812 09:45:56.220025 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:56.220030 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:56.220087 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:56.220161 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:56.221057 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:56.221647 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:56.222801 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:56.223023 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:56.223103 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:56.224750 (+ 1647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:45:56.224757 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:45:56.224800 (+ 43us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:45:56.224836 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:45:56.226876 (+ 2040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:56.227340 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:56.227346 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:56.227350 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:56.227476 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:45:56.227820 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:45:56.227829 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:45:56.227830 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:45:56.228261 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:45:56.228443 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0812 09:45:56.228769 (+ 326us) server_negotiation.cc:299] Negotiation successful
0812 09:45:56.229000 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":90,"threads_started":1}
Aug 12 09:45:56 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.3@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 09:45:56 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.3@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 09:45:56.230391 28497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:56.225199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:40203 (local address 127.13.99.3:49339)
0812 09:45:56.225428 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:56.225440 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:56.225522 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:56.225745 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:56.225748 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:56.225905 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:45:56.225994 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:56.225999 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:56.226747 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:56.226751 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:56.227296 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:56.227304 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:56.227813 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:56.227817 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:56.227920 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:56.228381 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:45:56.228395 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:45:56.230286 (+ 1891us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.99.62:40203: Server oryx/127.13.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164}
W20260812 09:45:56.230510 28035 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 09:45:56.225312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.3:49339 (local address 127.13.99.62:40203)
0812 09:45:56.225408 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:56.225412 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:56.225483 (+ 71us) server_negotiation.cc:407] Connection header received
0812 09:45:56.225599 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:56.225602 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:56.225638 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:56.225728 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:56.226113 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:56.226654 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:56.227422 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:56.227651 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:56.227763 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:56.230396 (+ 2633us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.99.3:49339: BlockingRecv error: recv got EOF from 127.13.99.3:49339 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Aug 12 09:45:56 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.3@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 09:45:56 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.3@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 09:45:56.235805 28497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:56.231086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:40203 (local address 127.13.99.3:55861)
0812 09:45:56.231235 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:56.231248 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:56.231317 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:56.231598 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:56.231601 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:56.231763 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:45:56.231831 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:56.231836 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:56.232667 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:56.232670 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:56.233163 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:56.233171 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:56.233558 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:56.233560 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:56.233615 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:56.234000 (+ 385us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:45:56.234011 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:45:56.235725 (+ 1714us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.99.62:40203: Server oryx/127.13.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":89}
W20260812 09:45:56.235966 28035 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 09:45:56.231160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.3:55861 (local address 127.13.99.62:40203)
0812 09:45:56.231304 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:56.231308 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:56.231319 (+ 11us) server_negotiation.cc:407] Connection header received
0812 09:45:56.231397 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:56.231399 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:56.231461 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:56.231549 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:56.231939 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:56.232571 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:56.233260 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:56.233465 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:56.233542 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:56.235837 (+ 2295us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.99.3:55861: BlockingRecv error: recv got EOF from 127.13.99.3:55861 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260812 09:45:56.236176 28495 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 12 09:45:56 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.1@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 09:45:56 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.1@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 09:45:56.849342 28502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:56.843702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:40203 (local address 127.13.99.1:50813)
0812 09:45:56.844014 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:56.844027 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:56.844112 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:56.844364 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:56.844367 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:56.844613 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:45:56.844735 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:56.844741 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:56.845492 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:56.845495 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:56.846073 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:56.846081 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:56.846486 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:56.846488 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:56.846560 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:56.847134 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:45:56.847147 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:45:56.849173 (+ 2026us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.99.62:40203: Server oryx/127.13.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":114,"threads_started":1}
W20260812 09:45:56.849501 28504 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 09:45:56.843792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.1:50813 (local address 127.13.99.62:40203)
0812 09:45:56.844058 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:56.844061 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:56.844075 (+ 14us) server_negotiation.cc:407] Connection header received
0812 09:45:56.844204 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:56.844206 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:56.844250 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:56.844328 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:56.844837 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:56.845394 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:56.846173 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:56.846388 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:56.846463 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:56.849340 (+ 2877us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.99.1:50813: BlockingRecv error: recv got EOF from 127.13.99.1:50813 (error 108)
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":108,"threads_started":1}
Aug 12 09:45:57 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.2@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 09:45:57 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.2@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 09:45:57.048167 28506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:57.042020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:40203 (local address 127.13.99.2:55709)
0812 09:45:57.042371 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:57.042388 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:57.042551 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:57.042842 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:57.042845 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:57.043062 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:45:57.043163 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:57.043168 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:57.043937 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:57.043940 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:57.044609 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:57.044619 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:57.045071 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:57.045074 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:57.045164 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:57.045732 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:45:57.045747 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:45:57.047986 (+ 2239us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.99.62:40203: Server oryx/127.13.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":90,"threads_started":1}
W20260812 09:45:57.048225 28504 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 09:45:57.042163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.2:55709 (local address 127.13.99.62:40203)
0812 09:45:57.042347 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:57.042352 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:57.042477 (+ 125us) server_negotiation.cc:407] Connection header received
0812 09:45:57.042679 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:57.042682 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:57.042729 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:57.042819 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:57.043281 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:57.043823 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:57.044727 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:57.044950 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:57.045069 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:57.048112 (+ 3043us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.99.2:55709: BlockingRecv error: recv got EOF from 127.13.99.2:55709 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Aug 12 09:45:57 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.3@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 09:45:57 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.3@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 09:45:57.242260 28510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:57.236780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:40203 (local address 127.13.99.3:47373)
0812 09:45:57.237080 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:57.237094 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:57.237195 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:57.237421 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:57.237424 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:57.237606 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:45:57.237706 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:57.237711 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:57.238429 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:57.238432 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:57.239017 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:57.239025 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:57.239469 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:57.239473 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:57.239543 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:57.240149 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:45:57.240162 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:45:57.242129 (+ 1967us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.99.62:40203: Server oryx/127.13.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":89,"threads_started":1}
W20260812 09:45:57.242384 28504 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 09:45:57.236866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.3:47373 (local address 127.13.99.62:40203)
0812 09:45:57.237012 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:57.237016 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:57.237167 (+ 151us) server_negotiation.cc:407] Connection header received
0812 09:45:57.237264 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:57.237266 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:57.237312 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:57.237380 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:57.237810 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:57.238331 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:57.239121 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:57.239336 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:57.239414 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:57.242256 (+ 2842us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.99.3:47373: BlockingRecv error: recv got EOF from 127.13.99.3:47373 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 12 09:45:57 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.1@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 09:45:57 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.1@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 09:45:57.856405 28512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:57.850280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:40203 (local address 127.13.99.1:56569)
0812 09:45:57.850602 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:57.850616 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:57.850745 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:57.851108 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:57.851111 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:57.851363 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:45:57.851509 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:57.851518 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:57.852531 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:57.852535 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:57.853263 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:57.853273 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:57.853712 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:57.853714 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:57.853774 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:57.854286 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:45:57.854297 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:45:57.856258 (+ 1961us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.99.62:40203: Server oryx/127.13.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":93,"threads_started":1}
W20260812 09:45:57.856621 28513 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 09:45:57.850429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.1:56569 (local address 127.13.99.62:40203)
0812 09:45:57.850684 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:57.850691 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:57.850708 (+ 17us) server_negotiation.cc:407] Connection header received
0812 09:45:57.850882 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:57.850888 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:57.850953 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:57.851050 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:57.851731 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:57.852374 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:57.853383 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:57.853610 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:57.853692 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:57.856420 (+ 2728us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.99.1:56569: BlockingRecv error: recv got EOF from 127.13.99.1:56569 (error 108)
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":63,"threads_started":1}
Aug 12 09:45:58 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.2@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 09:45:58 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.2@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 09:45:58.054208 28514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:58.048984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:40203 (local address 127.13.99.2:53341)
0812 09:45:58.049274 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:58.049288 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:58.049389 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:58.049626 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:58.049628 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:58.049812 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:45:58.049912 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:58.049917 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:58.050638 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:58.050641 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:58.051197 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:58.051205 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:58.051613 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:58.051616 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:58.051678 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:58.052159 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:45:58.052170 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:45:58.054070 (+ 1900us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.99.62:40203: Server oryx/127.13.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":121,"threads_started":1}
W20260812 09:45:58.054327 28513 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 09:45:58.049056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.2:53341 (local address 127.13.99.62:40203)
0812 09:45:58.049216 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:58.049221 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:58.049361 (+ 140us) server_negotiation.cc:407] Connection header received
0812 09:45:58.049472 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:58.049474 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:58.049520 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:58.049586 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:58.050018 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:58.050546 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:58.051299 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:58.051517 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:58.051593 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:58.054211 (+ 2618us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.99.2:53341: BlockingRecv error: recv got EOF from 127.13.99.2:53341 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 12 09:45:58 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.3@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 09:45:58 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.3@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 09:45:58.249148 28515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:58.243176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:40203 (local address 127.13.99.3:53847)
0812 09:45:58.243506 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:58.243529 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:58.243634 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:58.244048 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:58.244050 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:58.244296 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:45:58.244406 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:58.244412 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:58.245237 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:58.245242 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:58.245945 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:58.245956 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:58.246419 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:58.246422 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:58.246511 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:58.247093 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:45:58.247106 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:45:58.248960 (+ 1854us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.99.62:40203: Server oryx/127.13.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":87,"threads_started":1}
W20260812 09:45:58.249205 28513 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 09:45:58.243393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.3:53847 (local address 127.13.99.62:40203)
0812 09:45:58.243518 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:58.243522 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:58.243612 (+ 90us) server_negotiation.cc:407] Connection header received
0812 09:45:58.243843 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:58.243846 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:58.243899 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:58.243999 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:58.244565 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:58.245114 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:58.246072 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:58.246300 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:58.246415 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:58.249089 (+ 2674us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.99.3:53847: BlockingRecv error: recv got EOF from 127.13.99.3:53847 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Aug 12 09:45:58 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.1@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 09:45:58 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.1@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 09:45:58.862996 28519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:58.857278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:40203 (local address 127.13.99.1:52259)
0812 09:45:58.857557 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:58.857572 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:58.857670 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:58.857920 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:58.857922 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:58.858110 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:45:58.858210 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:58.858215 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:58.858982 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:58.858988 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:58.859797 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:58.859813 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:58.860427 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:58.860430 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:58.860519 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:58.861093 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:45:58.861106 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:45:58.862878 (+ 1772us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.99.62:40203: Server oryx/127.13.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":88,"threads_started":1}
W20260812 09:45:58.863204 28520 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 09:45:58.857341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.1:52259 (local address 127.13.99.62:40203)
0812 09:45:58.857583 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:58.857590 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:58.857624 (+ 34us) server_negotiation.cc:407] Connection header received
0812 09:45:58.857760 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:58.857762 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:58.857808 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:58.857885 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:58.858319 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:58.858847 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:58.859966 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:58.860316 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:58.860402 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:58.863020 (+ 2618us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.99.1:52259: BlockingRecv error: recv got EOF from 127.13.99.1:52259 (error 108)
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":73,"threads_started":1}
Aug 12 09:45:59 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.2@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 09:45:59 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.2@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 09:45:59.061004 28522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:59.055097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:40203 (local address 127.13.99.2:47933)
0812 09:45:59.055400 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:59.055413 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:59.055575 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:59.055908 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:59.055911 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:59.056123 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:45:59.056222 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:59.056228 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:59.057073 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:59.057079 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:59.057644 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:59.057653 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:59.058131 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:59.058135 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:59.058228 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:59.058765 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:45:59.058777 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:45:59.060793 (+ 2016us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.99.62:40203: Server oryx/127.13.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":99,"threads_started":1}
W20260812 09:45:59.061036 28520 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 09:45:59.055176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.2:47933 (local address 127.13.99.62:40203)
0812 09:45:59.055358 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:59.055362 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:59.055486 (+ 124us) server_negotiation.cc:407] Connection header received
0812 09:45:59.055695 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:59.055698 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:59.055749 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:59.055854 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:59.056353 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:59.056958 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:59.057775 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:59.058012 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:59.058127 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:59.060930 (+ 2803us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.99.2:47933: BlockingRecv error: recv got EOF from 127.13.99.2:47933 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Aug 12 09:45:59 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.3@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 09:45:59 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.3@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 09:45:59.255616 28523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:59.249945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:40203 (local address 127.13.99.3:44391)
0812 09:45:59.250237 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:59.250251 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:59.250355 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:59.250632 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:59.250638 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:59.250845 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:45:59.250953 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:59.250960 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:59.251983 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:59.251986 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:59.252614 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:59.252623 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:59.253023 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:59.253025 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:59.253087 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:59.253567 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:45:59.253579 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:45:59.255470 (+ 1891us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.99.62:40203: Server oryx/127.13.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":101,"threads_started":1}
W20260812 09:45:59.255740 28520 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 09:45:59.250061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.3:44391 (local address 127.13.99.62:40203)
0812 09:45:59.250244 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:59.250249 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:59.250326 (+ 77us) server_negotiation.cc:407] Connection header received
0812 09:45:59.250439 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:59.250441 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:59.250493 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:59.250570 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:59.251044 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:59.251835 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:59.252717 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:59.252926 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:59.253005 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:59.255603 (+ 2598us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.99.3:44391: BlockingRecv error: recv got EOF from 127.13.99.3:44391 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Aug 12 09:45:59 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.1@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 09:45:59 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.1@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 09:45:59.873092 28524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:45:59.863946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:40203 (local address 127.13.99.1:40499)
0812 09:45:59.864363 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0812 09:45:59.864376 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 09:45:59.864502 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:45:59.864748 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:45:59.864750 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:45:59.864962 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:45:59.865064 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:59.865069 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:59.865883 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:59.865886 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:59.866461 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:45:59.866469 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:45:59.866950 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:45:59.866953 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:45:59.867016 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:59.870962 (+ 3946us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:45:59.870976 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:45:59.872970 (+ 1994us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.99.62:40203: Server oryx/127.13.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":96,"threads_started":1}
W20260812 09:45:59.873310 28525 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 09:45:59.863993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.1:40499 (local address 127.13.99.62:40203)
0812 09:45:59.864330 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0812 09:45:59.864337 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 09:45:59.864446 (+ 109us) server_negotiation.cc:407] Connection header received
0812 09:45:59.864583 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:45:59.864586 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:45:59.864632 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:45:59.864710 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:45:59.865206 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:59.865777 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:59.866590 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:45:59.866824 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:45:59.866913 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:45:59.873155 (+ 6242us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.99.1:40499: BlockingRecv error: recv got EOF from 127.13.99.1:40499 (error 108)
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":129,"threads_started":1}
Aug 12 09:46:00 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.2@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 09:46:00 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.2@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 09:46:00.067492 28526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:46:00.061840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:40203 (local address 127.13.99.2:57483)
0812 09:46:00.062160 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0812 09:46:00.062176 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 09:46:00.062323 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:46:00.062601 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:46:00.062604 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:46:00.062831 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:46:00.062930 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:46:00.062936 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:46:00.063703 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:46:00.063708 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:46:00.064293 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:46:00.064309 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:46:00.064821 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:46:00.064825 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:46:00.064919 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:46:00.065430 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:46:00.065443 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:46:00.067307 (+ 1864us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.99.62:40203: Server oryx/127.13.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":91,"threads_started":1}
W20260812 09:46:00.067552 28525 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 09:46:00.061964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.2:57483 (local address 127.13.99.62:40203)
0812 09:46:00.062136 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0812 09:46:00.062141 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:46:00.062259 (+ 118us) server_negotiation.cc:407] Connection header received
0812 09:46:00.062431 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:46:00.062434 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:46:00.062487 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:46:00.062578 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:46:00.063052 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:46:00.063587 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:46:00.064422 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:46:00.064703 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:46:00.064823 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:46:00.067441 (+ 2618us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.99.2:57483: BlockingRecv error: recv got EOF from 127.13.99.2:57483 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 12 09:46:00 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.3@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 09:46:00 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.3@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 09:46:00.262250 28527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:46:00.256412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:40203 (local address 127.13.99.3:42477)
0812 09:46:00.256796 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0812 09:46:00.256813 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 09:46:00.256963 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:46:00.257258 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:46:00.257261 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:46:00.257485 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:46:00.257580 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:46:00.257586 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:46:00.258428 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:46:00.258431 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:46:00.259171 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:46:00.259183 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:46:00.259646 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:46:00.259649 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:46:00.259714 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:46:00.260211 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:46:00.260223 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:46:00.262079 (+ 1856us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.99.62:40203: Server oryx/127.13.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":146,"threads_started":1}
W20260812 09:46:00.262334 28525 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 09:46:00.256579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.3:42477 (local address 127.13.99.62:40203)
0812 09:46:00.256738 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0812 09:46:00.256744 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 09:46:00.256903 (+ 159us) server_negotiation.cc:407] Connection header received
0812 09:46:00.257079 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:46:00.257082 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:46:00.257145 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:46:00.257229 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:46:00.257696 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:46:00.258314 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:46:00.259293 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:46:00.259541 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:46:00.259629 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:46:00.262214 (+ 2585us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.99.3:42477: BlockingRecv error: recv got EOF from 127.13.99.3:42477 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 12 09:46:00 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.1@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 09:46:00 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.1@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 09:46:00.880064 28528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:46:00.874065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:40203 (local address 127.13.99.1:43325)
0812 09:46:00.874387 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0812 09:46:00.874401 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 09:46:00.874502 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:46:00.874756 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:46:00.874759 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:46:00.874954 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:46:00.875053 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:46:00.875058 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:46:00.875836 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:46:00.875839 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:46:00.876403 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:46:00.876410 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:46:00.877019 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:46:00.877022 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:46:00.877151 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:46:00.877689 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:46:00.877701 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:46:00.879889 (+ 2188us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.99.62:40203: Server oryx/127.13.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":92,"threads_started":1}
W20260812 09:46:00.880296 28529 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 09:46:00.874132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.1:43325 (local address 127.13.99.62:40203)
0812 09:46:00.874443 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0812 09:46:00.874446 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 09:46:00.874461 (+ 15us) server_negotiation.cc:407] Connection header received
0812 09:46:00.874591 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:46:00.874594 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:46:00.874639 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:46:00.874721 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:46:00.875185 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:46:00.875739 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:46:00.876592 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:46:00.876913 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:46:00.876998 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:46:00.880175 (+ 3177us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.99.1:43325: BlockingRecv error: recv got EOF from 127.13.99.1:43325 (error 108)
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":115,"threads_started":1}
Aug 12 09:46:01 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.2@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 09:46:01 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.2@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 09:46:01.073887 28532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:46:01.068310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:40203 (local address 127.13.99.2:36561)
0812 09:46:01.068628 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0812 09:46:01.068641 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 09:46:01.068740 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:46:01.069005 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:46:01.069008 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:46:01.069199 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:46:01.069300 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:46:01.069306 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:46:01.070110 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:46:01.070113 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:46:01.070692 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:46:01.070700 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:46:01.071153 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:46:01.071155 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:46:01.071220 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:46:01.071805 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:46:01.071817 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:46:01.073721 (+ 1904us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.99.62:40203: Server oryx/127.13.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":120,"threads_started":1}
W20260812 09:46:01.074049 28529 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 09:46:01.068456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.2:36561 (local address 127.13.99.62:40203)
0812 09:46:01.068624 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0812 09:46:01.068628 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:46:01.068694 (+ 66us) server_negotiation.cc:407] Connection header received
0812 09:46:01.068807 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:46:01.068809 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:46:01.068850 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:46:01.068919 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:46:01.069443 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:46:01.069997 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:46:01.070815 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:46:01.071047 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:46:01.071132 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:46:01.073898 (+ 2766us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.99.2:36561: BlockingRecv error: recv got EOF from 127.13.99.2:36561 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Aug 12 09:46:01 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.3@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 09:46:01 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.3@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 09:46:01.271756 28535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:46:01.264960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:40203 (local address 127.13.99.3:49587)
0812 09:46:01.265264 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0812 09:46:01.265284 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 09:46:01.265451 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:46:01.265760 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:46:01.265763 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:46:01.265969 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:46:01.266115 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:46:01.266127 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:46:01.267379 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:46:01.267384 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:46:01.268392 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:46:01.268408 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:46:01.268953 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:46:01.268957 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:46:01.269051 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:46:01.269655 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:46:01.269677 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:46:01.271564 (+ 1887us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.99.62:40203: Server oryx/127.13.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":94,"threads_started":1}
W20260812 09:46:01.271833 28529 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 09:46:01.265111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.3:49587 (local address 127.13.99.62:40203)
0812 09:46:01.265263 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0812 09:46:01.265268 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:46:01.265396 (+ 128us) server_negotiation.cc:407] Connection header received
0812 09:46:01.265572 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:46:01.265577 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:46:01.265635 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:46:01.265714 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:46:01.266568 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:46:01.267241 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:46:01.268545 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:46:01.268804 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:46:01.268935 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:46:01.271708 (+ 2773us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.99.3:49587: BlockingRecv error: recv got EOF from 127.13.99.3:49587 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 12 09:46:01 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.1@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 09:46:01 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.1@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 09:46:01.886345 28536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:46:01.881026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:40203 (local address 127.13.99.1:40263)
0812 09:46:01.881306 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0812 09:46:01.881320 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 09:46:01.881414 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:46:01.881661 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:46:01.881664 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:46:01.881854 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:46:01.881952 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:46:01.881957 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:46:01.882700 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:46:01.882702 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:46:01.883257 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:46:01.883264 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:46:01.883695 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:46:01.883697 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:46:01.883762 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:46:01.884322 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:46:01.884335 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:46:01.886209 (+ 1874us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.99.62:40203: Server oryx/127.13.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":73,"threads_started":1}
W20260812 09:46:01.886519 28538 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 09:46:01.881116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.1:40263 (local address 127.13.99.62:40203)
0812 09:46:01.881378 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0812 09:46:01.881382 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:46:01.881395 (+ 13us) server_negotiation.cc:407] Connection header received
0812 09:46:01.881501 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:46:01.881504 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:46:01.881548 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:46:01.881627 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:46:01.882053 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:46:01.882599 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:46:01.883364 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:46:01.883596 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:46:01.883678 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:46:01.886381 (+ 2703us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.99.1:40263: BlockingRecv error: recv got EOF from 127.13.99.1:40263 (error 108)
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":80,"threads_started":1}
Aug 12 09:46:02 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.2@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 09:46:02 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.2@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 09:46:02.080277 28541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:46:02.074778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:40203 (local address 127.13.99.2:46181)
0812 09:46:02.075095 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0812 09:46:02.075115 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 09:46:02.075241 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:46:02.075481 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:46:02.075483 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:46:02.075685 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:46:02.075788 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:46:02.075793 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:46:02.076548 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:46:02.076551 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:46:02.077148 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:46:02.077164 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:46:02.077572 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:46:02.077574 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:46:02.077637 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:46:02.078132 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:46:02.078144 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:46:02.080159 (+ 2015us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.99.62:40203: Server oryx/127.13.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":118,"threads_started":1}
W20260812 09:46:02.080406 28538 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 09:46:02.074858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.2:46181 (local address 127.13.99.62:40203)
0812 09:46:02.075023 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0812 09:46:02.075028 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:46:02.075195 (+ 167us) server_negotiation.cc:407] Connection header received
0812 09:46:02.075324 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:46:02.075326 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:46:02.075374 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:46:02.075443 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:46:02.075885 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:46:02.076400 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:46:02.077261 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:46:02.077478 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:46:02.077557 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:46:02.080292 (+ 2735us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.99.2:46181: BlockingRecv error: recv got EOF from 127.13.99.2:46181 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 12 09:46:02 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.3@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 09:46:02 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.3@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 09:46:02.278353 28542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:46:02.272658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:40203 (local address 127.13.99.3:59767)
0812 09:46:02.273027 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0812 09:46:02.273042 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 09:46:02.273177 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:46:02.273463 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:46:02.273466 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:46:02.273686 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:46:02.273779 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:46:02.273785 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:46:02.274557 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:46:02.274563 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:46:02.275132 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:46:02.275140 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:46:02.275590 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:46:02.275593 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:46:02.275682 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:46:02.276243 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:46:02.276258 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:46:02.278238 (+ 1980us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.99.62:40203: Server oryx/127.13.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":119,"threads_started":1}
W20260812 09:46:02.278494 28538 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 09:46:02.272740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.3:59767 (local address 127.13.99.62:40203)
0812 09:46:02.272950 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0812 09:46:02.272955 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:46:02.273126 (+ 171us) server_negotiation.cc:407] Connection header received
0812 09:46:02.273296 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:46:02.273299 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:46:02.273349 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:46:02.273435 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:46:02.273901 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:46:02.274438 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:46:02.275250 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:46:02.275476 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:46:02.275586 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:46:02.278372 (+ 2786us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.99.3:59767: BlockingRecv error: recv got EOF from 127.13.99.3:59767 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Aug 12 09:46:02 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.1@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 09:46:02 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.1@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 09:46:02.892956 28543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:46:02.887218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:40203 (local address 127.13.99.1:37941)
0812 09:46:02.887554 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0812 09:46:02.887567 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 09:46:02.887674 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:46:02.887939 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:46:02.887942 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:46:02.888131 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:46:02.888228 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:46:02.888234 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:46:02.889018 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:46:02.889022 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:46:02.889626 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:46:02.889636 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:46:02.890221 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:46:02.890225 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:46:02.890324 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:46:02.890912 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:46:02.890925 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:46:02.892838 (+ 1913us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.99.62:40203: Server oryx/127.13.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":100,"threads_started":1}
W20260812 09:46:02.893113 28544 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 09:46:02.887316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.1:37941 (local address 127.13.99.62:40203)
0812 09:46:02.887556 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0812 09:46:02.887560 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:46:02.887638 (+ 78us) server_negotiation.cc:407] Connection header received
0812 09:46:02.887761 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:46:02.887763 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:46:02.887811 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:46:02.887897 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:46:02.888330 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:46:02.888891 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:46:02.889752 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:46:02.890088 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:46:02.890211 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:46:02.892973 (+ 2762us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.99.1:37941: BlockingRecv error: recv got EOF from 127.13.99.1:37941 (error 108)
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":87,"threads_started":1}
Aug 12 09:46:03 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.2@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 09:46:03 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.2@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 09:46:03.086935 28545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:46:03.081199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:40203 (local address 127.13.99.2:54477)
0812 09:46:03.081567 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0812 09:46:03.081580 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 09:46:03.081710 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:46:03.082011 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:46:03.082014 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:46:03.082226 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:46:03.082319 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:46:03.082325 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:46:03.083084 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:46:03.083090 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:46:03.083662 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:46:03.083671 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:46:03.084199 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:46:03.084203 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:46:03.084308 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:46:03.084943 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:46:03.084956 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:46:03.086821 (+ 1865us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.99.62:40203: Server oryx/127.13.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":131,"threads_started":1}
W20260812 09:46:03.087096 28544 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 09:46:03.081256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.2:54477 (local address 127.13.99.62:40203)
0812 09:46:03.081407 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0812 09:46:03.081414 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 09:46:03.081660 (+ 246us) server_negotiation.cc:407] Connection header received
0812 09:46:03.081816 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:46:03.081820 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:46:03.081871 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:46:03.081957 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:46:03.082446 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:46:03.082973 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:46:03.083786 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:46:03.084049 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:46:03.084158 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:46:03.086965 (+ 2807us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.99.2:54477: BlockingRecv error: recv got EOF from 127.13.99.2:54477 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 12 09:46:03 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.3@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 09:46:03 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.3@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 09:46:03.285511 28548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:46:03.279252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:40203 (local address 127.13.99.3:57325)
0812 09:46:03.279618 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0812 09:46:03.279632 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 09:46:03.279767 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:46:03.280130 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:46:03.280133 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:46:03.280401 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:46:03.280569 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:46:03.280579 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:46:03.281599 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:46:03.281602 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:46:03.282193 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:46:03.282202 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:46:03.282664 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:46:03.282667 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:46:03.282739 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:46:03.283286 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:46:03.283298 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:46:03.285335 (+ 2037us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.99.62:40203: Server oryx/127.13.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":105,"threads_started":1}
W20260812 09:46:03.285593 28544 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 09:46:03.279300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.3:57325 (local address 127.13.99.62:40203)
0812 09:46:03.279455 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0812 09:46:03.279459 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:46:03.279709 (+ 250us) server_negotiation.cc:407] Connection header received
0812 09:46:03.279875 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:46:03.279879 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:46:03.279999 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:46:03.280096 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:46:03.280743 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:46:03.281498 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:46:03.282336 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:46:03.282555 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:46:03.282647 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:46:03.285475 (+ 2828us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.99.3:57325: BlockingRecv error: recv got EOF from 127.13.99.3:57325 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 12 09:46:03 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.1@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 09:46:03 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.1@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 09:46:03.899369 28549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:46:03.893856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:40203 (local address 127.13.99.1:57445)
0812 09:46:03.894152 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0812 09:46:03.894166 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 09:46:03.894291 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:46:03.894624 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:46:03.894633 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:46:03.894865 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:46:03.895018 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:46:03.895026 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:46:03.895800 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:46:03.895802 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:46:03.896354 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:46:03.896361 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:46:03.896879 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:46:03.896881 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:46:03.896944 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:46:03.897434 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:46:03.897446 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:46:03.899252 (+ 1806us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.99.62:40203: Server oryx/127.13.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":104,"threads_started":1}
W20260812 09:46:03.899539 28550 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 09:46:03.894029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.1:57445 (local address 127.13.99.62:40203)
0812 09:46:03.894255 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0812 09:46:03.894262 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 09:46:03.894284 (+ 22us) server_negotiation.cc:407] Connection header received
0812 09:46:03.894415 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:46:03.894425 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:46:03.894483 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:46:03.894586 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:46:03.895149 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:46:03.895707 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:46:03.896463 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:46:03.896772 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:46:03.896851 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:46:03.899386 (+ 2535us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.99.1:57445: BlockingRecv error: recv got EOF from 127.13.99.1:57445 (error 108)
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":71,"threads_started":1}
Aug 12 09:46:04 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.2@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 09:46:04 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.2@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 09:46:04.093555 28553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:46:04.087731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:40203 (local address 127.13.99.2:46471)
0812 09:46:04.088066 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0812 09:46:04.088080 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 09:46:04.088181 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:46:04.088423 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:46:04.088425 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:46:04.088639 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:46:04.088737 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:46:04.088743 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:46:04.089653 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:46:04.089657 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:46:04.090332 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:46:04.090343 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:46:04.090818 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:46:04.090827 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:46:04.090925 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:46:04.091575 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:46:04.091588 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:46:04.093440 (+ 1852us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.99.62:40203: Server oryx/127.13.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":90,"threads_started":1}
W20260812 09:46:04.093690 28550 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 09:46:04.087846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.2:46471 (local address 127.13.99.62:40203)
0812 09:46:04.087995 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0812 09:46:04.087999 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:46:04.088152 (+ 153us) server_negotiation.cc:407] Connection header received
0812 09:46:04.088267 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:46:04.088270 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:46:04.088315 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:46:04.088388 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:46:04.088834 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:46:04.089526 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:46:04.090484 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:46:04.090704 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:46:04.090784 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:46:04.093572 (+ 2788us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.99.2:46471: BlockingRecv error: recv got EOF from 127.13.99.2:46471 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 12 09:46:04 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.3@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 09:46:04 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.3@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 09:46:04.291962 28555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:46:04.286399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:40203 (local address 127.13.99.3:55265)
0812 09:46:04.286708 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0812 09:46:04.286722 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 09:46:04.286860 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:46:04.287148 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:46:04.287151 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:46:04.287366 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:46:04.287463 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:46:04.287469 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:46:04.288239 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:46:04.288243 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:46:04.288875 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:46:04.288885 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:46:04.289345 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:46:04.289348 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:46:04.289436 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:46:04.289998 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:46:04.290011 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:46:04.291791 (+ 1780us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.99.62:40203: Server oryx/127.13.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":113,"threads_started":1}
W20260812 09:46:04.292033 28550 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 09:46:04.286476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.3:55265 (local address 127.13.99.62:40203)
0812 09:46:04.286623 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0812 09:46:04.286628 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:46:04.286806 (+ 178us) server_negotiation.cc:407] Connection header received
0812 09:46:04.286971 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:46:04.286974 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:46:04.287030 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:46:04.287117 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:46:04.287586 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:46:04.288125 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:46:04.288999 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:46:04.289229 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:46:04.289339 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:46:04.291919 (+ 2580us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.99.3:55265: BlockingRecv error: recv got EOF from 127.13.99.3:55265 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 12 09:46:04 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.1@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 09:46:04 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.1@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 09:46:04.908797 28558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:46:04.900201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:40203 (local address 127.13.99.1:53373)
0812 09:46:04.900550 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0812 09:46:04.900568 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 09:46:04.900680 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:46:04.900983 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:46:04.900986 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:46:04.901235 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:46:04.901347 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:46:04.901354 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:46:04.902107 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:46:04.902110 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:46:04.902671 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:46:04.902679 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:46:04.903088 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:46:04.903091 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:46:04.903155 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:46:04.906705 (+ 3550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:46:04.906718 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:46:04.908680 (+ 1962us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.99.62:40203: Server oryx/127.13.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":96,"threads_started":1}
W20260812 09:46:04.908963 28559 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 09:46:04.900364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.1:53373 (local address 127.13.99.62:40203)
0812 09:46:04.900634 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0812 09:46:04.900639 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:46:04.900653 (+ 14us) server_negotiation.cc:407] Connection header received
0812 09:46:04.900776 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:46:04.900779 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:46:04.900832 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:46:04.900936 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:46:04.901459 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:46:04.902002 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:46:04.902767 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:46:04.902997 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:46:04.903070 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:46:04.908826 (+ 5756us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.99.1:53373: BlockingRecv error: recv got EOF from 127.13.99.1:53373 (error 108)
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":74,"threads_started":1}
Aug 12 09:46:05 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.2@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 09:46:05 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.2@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 09:46:05.100320 28560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:46:05.094507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:40203 (local address 127.13.99.2:47115)
0812 09:46:05.094856 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0812 09:46:05.094869 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 09:46:05.095013 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:46:05.095284 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:46:05.095287 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:46:05.095504 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:46:05.095602 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:46:05.095608 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:46:05.096509 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:46:05.096513 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:46:05.097165 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:46:05.097175 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:46:05.097632 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:46:05.097635 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:46:05.097723 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:46:05.098284 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:46:05.098302 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:46:05.100205 (+ 1903us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.99.62:40203: Server oryx/127.13.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":123,"threads_started":1}
W20260812 09:46:05.100620 28559 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 09:46:05.094594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.2:47115 (local address 127.13.99.62:40203)
0812 09:46:05.094776 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0812 09:46:05.094781 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:46:05.094954 (+ 173us) server_negotiation.cc:407] Connection header received
0812 09:46:05.095115 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:46:05.095119 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:46:05.095171 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:46:05.095261 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:46:05.095729 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:46:05.096365 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:46:05.097290 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:46:05.097519 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:46:05.097627 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:46:05.100350 (+ 2723us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.99.2:47115: BlockingRecv error: recv got EOF from 127.13.99.2:47115 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Aug 12 09:46:05 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.3@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 09:46:05 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.3@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 09:46:05.298470 28561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:46:05.292822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:40203 (local address 127.13.99.3:46683)
0812 09:46:05.293192 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0812 09:46:05.293206 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 09:46:05.293340 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:46:05.293620 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:46:05.293623 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:46:05.293834 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:46:05.293929 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:46:05.293934 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:46:05.294732 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:46:05.294737 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:46:05.295299 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:46:05.295307 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:46:05.295767 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:46:05.295771 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:46:05.295860 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:46:05.296379 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:46:05.296393 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:46:05.298299 (+ 1906us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.99.62:40203: Server oryx/127.13.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":103,"threads_started":1}
W20260812 09:46:05.298559 28559 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 09:46:05.292902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.3:46683 (local address 127.13.99.62:40203)
0812 09:46:05.293063 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0812 09:46:05.293070 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 09:46:05.293288 (+ 218us) server_negotiation.cc:407] Connection header received
0812 09:46:05.293441 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:46:05.293444 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:46:05.293503 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:46:05.293590 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:46:05.294068 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:46:05.294620 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:46:05.295420 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:46:05.295652 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:46:05.295761 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:46:05.298426 (+ 2665us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.99.3:46683: BlockingRecv error: recv got EOF from 127.13.99.3:46683 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Aug 12 09:46:05 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.1@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 09:46:05 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.1@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 09:46:05.915297 28562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:46:05.909738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:40203 (local address 127.13.99.1:56411)
0812 09:46:05.910022 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0812 09:46:05.910037 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 09:46:05.910130 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:46:05.910381 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:46:05.910384 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:46:05.910604 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:46:05.910708 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:46:05.910714 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:46:05.911551 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:46:05.911554 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:46:05.912115 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:46:05.912124 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:46:05.912603 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:46:05.912606 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:46:05.912683 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:46:05.913245 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:46:05.913259 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:46:05.915178 (+ 1919us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.99.62:40203: Server oryx/127.13.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":98,"threads_started":1}
W20260812 09:46:05.915452 28563 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 09:46:05.909862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.1:56411 (local address 127.13.99.62:40203)
0812 09:46:05.910093 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0812 09:46:05.910097 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:46:05.910110 (+ 13us) server_negotiation.cc:407] Connection header received
0812 09:46:05.910216 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:46:05.910218 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:46:05.910263 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:46:05.910348 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:46:05.910861 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:46:05.911432 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:46:05.912243 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:46:05.912496 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:46:05.912590 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:46:05.915321 (+ 2731us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.99.1:56411: BlockingRecv error: recv got EOF from 127.13.99.1:56411 (error 108)
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":85,"threads_started":1}
Aug 12 09:46:06 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.2@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 09:46:06 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.2@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 09:46:06.106560 28564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:46:06.101139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:40203 (local address 127.13.99.2:33987)
0812 09:46:06.101427 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0812 09:46:06.101441 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 09:46:06.101542 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:46:06.101802 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:46:06.101805 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:46:06.101993 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:46:06.102122 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:46:06.102128 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:46:06.102862 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:46:06.102865 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:46:06.103429 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:46:06.103436 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:46:06.103840 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:46:06.103843 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:46:06.103908 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:46:06.104429 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:46:06.104441 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:46:06.106361 (+ 1920us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.99.62:40203: Server oryx/127.13.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":108,"threads_started":1}
W20260812 09:46:06.106599 28563 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 09:46:06.101209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.2:33987 (local address 127.13.99.62:40203)
0812 09:46:06.101345 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0812 09:46:06.101349 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:46:06.101512 (+ 163us) server_negotiation.cc:407] Connection header received
0812 09:46:06.101610 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:46:06.101612 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:46:06.101655 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:46:06.101723 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:46:06.102230 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:46:06.102765 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:46:06.103531 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:46:06.103744 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:46:06.103821 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:46:06.106503 (+ 2682us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.99.2:33987: BlockingRecv error: recv got EOF from 127.13.99.2:33987 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Aug 12 09:46:06 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.3@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 09:46:06 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.3@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 09:46:06.304740 28565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:46:06.299358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:40203 (local address 127.13.99.3:43819)
0812 09:46:06.299661 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0812 09:46:06.299676 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 09:46:06.299776 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:46:06.300094 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:46:06.300096 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:46:06.300280 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:46:06.300378 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:46:06.300383 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:46:06.301198 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:46:06.301201 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:46:06.301770 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:46:06.301777 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:46:06.302198 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:46:06.302200 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:46:06.302261 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:46:06.302759 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:46:06.302770 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:46:06.304599 (+ 1829us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.99.62:40203: Server oryx/127.13.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":78,"threads_started":1}
W20260812 09:46:06.305042 28563 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 09:46:06.299530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.3:43819 (local address 127.13.99.62:40203)
0812 09:46:06.299691 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0812 09:46:06.299694 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 09:46:06.299739 (+ 45us) server_negotiation.cc:407] Connection header received
0812 09:46:06.299917 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:46:06.299920 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:46:06.299971 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:46:06.300052 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:46:06.300523 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:46:06.301090 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:46:06.301884 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:46:06.302100 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:46:06.302180 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:46:06.304750 (+ 2570us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.99.3:43819: BlockingRecv error: recv got EOF from 127.13.99.3:43819 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Aug 12 09:46:06 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.1@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 09:46:06 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.1@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 09:46:06.921821 28566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:46:06.916184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:40203 (local address 127.13.99.1:39307)
0812 09:46:06.916468 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0812 09:46:06.916499 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0812 09:46:06.916596 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:46:06.916859 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:46:06.916861 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:46:06.917048 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:46:06.917147 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:46:06.917151 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:46:06.918029 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:46:06.918035 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:46:06.918652 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:46:06.918663 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:46:06.919179 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:46:06.919183 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:46:06.919282 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:46:06.919806 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:46:06.919818 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:46:06.921650 (+ 1832us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.99.62:40203: Server oryx/127.13.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":102,"threads_started":1}
W20260812 09:46:06.921904 28567 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 09:46:06.916247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.1:39307 (local address 127.13.99.62:40203)
0812 09:46:06.916556 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0812 09:46:06.916560 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:46:06.916576 (+ 16us) server_negotiation.cc:407] Connection header received
0812 09:46:06.916683 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:46:06.916686 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:46:06.916732 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:46:06.916818 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:46:06.917247 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:46:06.917905 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:46:06.918791 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:46:06.919050 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:46:06.919160 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:46:06.921777 (+ 2617us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.99.1:39307: BlockingRecv error: recv got EOF from 127.13.99.1:39307 (error 108)
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":91,"threads_started":1}
Aug 12 09:46:07 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.2@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 09:46:07 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.2@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 09:46:07.112622 28569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:46:07.107407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:40203 (local address 127.13.99.2:57995)
0812 09:46:07.107698 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0812 09:46:07.107714 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 09:46:07.107821 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:46:07.108053 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:46:07.108055 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:46:07.108237 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:46:07.108334 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:46:07.108339 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:46:07.109089 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:46:07.109091 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:46:07.109637 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:46:07.109644 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:46:07.110074 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:46:07.110077 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:46:07.110139 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:46:07.110652 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:46:07.110664 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:46:07.112505 (+ 1841us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.99.62:40203: Server oryx/127.13.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":120,"threads_started":1}
W20260812 09:46:07.112857 28567 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 09:46:07.107472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.2:57995 (local address 127.13.99.62:40203)
0812 09:46:07.107628 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0812 09:46:07.107633 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:46:07.107788 (+ 155us) server_negotiation.cc:407] Connection header received
0812 09:46:07.107901 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:46:07.107904 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:46:07.107950 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:46:07.108018 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:46:07.108434 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:46:07.109003 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:46:07.109739 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:46:07.109979 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:46:07.110059 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:46:07.112733 (+ 2674us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.99.2:57995: BlockingRecv error: recv got EOF from 127.13.99.2:57995 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 12 09:46:07 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.3@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 09:46:07 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.3@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 09:46:07.310949 28570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:46:07.305546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:40203 (local address 127.13.99.3:47391)
0812 09:46:07.305867 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0812 09:46:07.305882 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 09:46:07.305994 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:46:07.306256 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:46:07.306258 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:46:07.306451 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:46:07.306555 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:46:07.306561 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:46:07.307339 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:46:07.307342 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:46:07.307911 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:46:07.307920 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:46:07.308365 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:46:07.308368 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:46:07.308439 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:46:07.309054 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:46:07.309066 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:46:07.310832 (+ 1766us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.99.62:40203: Server oryx/127.13.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":125,"threads_started":1}
W20260812 09:46:07.311131 28567 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 09:46:07.305621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.3:47391 (local address 127.13.99.62:40203)
0812 09:46:07.305777 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0812 09:46:07.305782 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:46:07.305942 (+ 160us) server_negotiation.cc:407] Connection header received
0812 09:46:07.306076 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:46:07.306078 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:46:07.306129 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:46:07.306198 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:46:07.306664 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:46:07.307211 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:46:07.308045 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:46:07.308258 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:46:07.308336 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:46:07.310984 (+ 2648us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.99.3:47391: BlockingRecv error: recv got EOF from 127.13.99.3:47391 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 12 09:46:07 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.1@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 09:46:07 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.1@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 09:46:07.928422 28572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:46:07.922694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:40203 (local address 127.13.99.1:47331)
0812 09:46:07.922996 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0812 09:46:07.923011 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 09:46:07.923113 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:46:07.923354 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:46:07.923357 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:46:07.923542 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:46:07.923641 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:46:07.923645 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:46:07.924411 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:46:07.924417 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:46:07.925095 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:46:07.925105 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:46:07.925587 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:46:07.925591 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:46:07.925672 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:46:07.926267 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:46:07.926295 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:46:07.928254 (+ 1959us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.99.62:40203: Server oryx/127.13.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":73,"threads_started":1}
W20260812 09:46:07.928530 28573 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 09:46:07.922761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.1:47331 (local address 127.13.99.62:40203)
0812 09:46:07.923023 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0812 09:46:07.923048 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0812 09:46:07.923061 (+ 13us) server_negotiation.cc:407] Connection header received
0812 09:46:07.923194 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:46:07.923196 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:46:07.923239 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:46:07.923308 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:46:07.923743 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:46:07.924291 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:46:07.925250 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:46:07.925475 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:46:07.925584 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:46:07.928377 (+ 2793us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.99.1:47331: BlockingRecv error: recv got EOF from 127.13.99.1:47331 (error 108)
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":114,"threads_started":1}
Aug 12 09:46:08 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.2@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 09:46:08 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.2@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 09:46:08.118733 28574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:46:08.113443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:40203 (local address 127.13.99.2:51311)
0812 09:46:08.113722 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0812 09:46:08.113736 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 09:46:08.113834 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:46:08.114076 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:46:08.114078 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:46:08.114262 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:46:08.114362 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:46:08.114368 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:46:08.115120 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:46:08.115123 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:46:08.115680 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:46:08.115687 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:46:08.116094 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:46:08.116097 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:46:08.116161 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:46:08.116760 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:46:08.116773 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:46:08.118617 (+ 1844us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.99.62:40203: Server oryx/127.13.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":95,"threads_started":1}
W20260812 09:46:08.118882 28573 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 09:46:08.113504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.2:51311 (local address 127.13.99.62:40203)
0812 09:46:08.113676 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0812 09:46:08.113680 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:46:08.113804 (+ 124us) server_negotiation.cc:407] Connection header received
0812 09:46:08.113919 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:46:08.113921 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:46:08.113967 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:46:08.114031 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:46:08.114462 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:46:08.115015 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:46:08.115776 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:46:08.116002 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:46:08.116079 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:46:08.118758 (+ 2679us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.99.2:51311: BlockingRecv error: recv got EOF from 127.13.99.2:51311 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Aug 12 09:46:08 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.3@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 09:46:08 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.3@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 09:46:08.317837 28575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:46:08.311760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:40203 (local address 127.13.99.3:32849)
0812 09:46:08.312112 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0812 09:46:08.312128 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 09:46:08.312270 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:46:08.312586 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:46:08.312589 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:46:08.312810 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:46:08.312943 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:46:08.312949 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:46:08.313718 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:46:08.313723 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:46:08.314286 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:46:08.314295 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:46:08.314778 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:46:08.314782 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:46:08.314866 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:46:08.315437 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:46:08.315450 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:46:08.317675 (+ 2225us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.99.62:40203: Server oryx/127.13.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":88,"threads_started":1}
W20260812 09:46:08.318141 28573 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 09:46:08.311883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.3:32849 (local address 127.13.99.62:40203)
0812 09:46:08.312073 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0812 09:46:08.312078 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:46:08.312216 (+ 138us) server_negotiation.cc:407] Connection header received
0812 09:46:08.312379 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:46:08.312386 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:46:08.312443 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:46:08.312564 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:46:08.313063 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:46:08.313602 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:46:08.314420 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:46:08.314661 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:46:08.314770 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:46:08.317846 (+ 3076us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.99.3:32849: BlockingRecv error: recv got EOF from 127.13.99.3:32849 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Aug 12 09:46:08 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.1@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 09:46:08 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.1@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 09:46:08.934381 28577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:46:08.929209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:40203 (local address 127.13.99.1:52121)
0812 09:46:08.929473 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0812 09:46:08.929487 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 09:46:08.929578 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:46:08.929818 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:46:08.929820 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:46:08.930002 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:46:08.930099 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:46:08.930104 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:46:08.930921 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:46:08.930923 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:46:08.931475 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:46:08.931483 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:46:08.931944 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:46:08.931947 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:46:08.932008 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:46:08.932536 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:46:08.932548 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:46:08.934253 (+ 1705us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.99.62:40203: Server oryx/127.13.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":89,"threads_started":1}
W20260812 09:46:08.934521 28578 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 09:46:08.929279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.1:52121 (local address 127.13.99.62:40203)
0812 09:46:08.929515 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0812 09:46:08.929519 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:46:08.929532 (+ 13us) server_negotiation.cc:407] Connection header received
0812 09:46:08.929661 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:46:08.929663 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:46:08.929708 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:46:08.929788 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:46:08.930232 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:46:08.930785 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:46:08.931588 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:46:08.931814 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:46:08.931929 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:46:08.934386 (+ 2457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.99.1:52121: BlockingRecv error: recv got EOF from 127.13.99.1:52121 (error 108)
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":83,"threads_started":1}
Aug 12 09:46:09 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.2@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 09:46:09 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.2@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 09:46:09.125134 28579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:46:09.119590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:40203 (local address 127.13.99.2:39873)
0812 09:46:09.119860 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0812 09:46:09.119880 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 09:46:09.119973 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:46:09.120262 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:46:09.120265 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:46:09.120531 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:46:09.120652 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:46:09.120659 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:46:09.121380 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:46:09.121383 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:46:09.121936 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:46:09.121944 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:46:09.122422 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:46:09.122424 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:46:09.122488 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:46:09.123071 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:46:09.123085 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:46:09.125016 (+ 1931us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.99.62:40203: Server oryx/127.13.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":98,"threads_started":1}
W20260812 09:46:09.125319 28578 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 09:46:09.119778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.2:39873 (local address 127.13.99.62:40203)
0812 09:46:09.119927 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0812 09:46:09.119931 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:46:09.119947 (+ 16us) server_negotiation.cc:407] Connection header received
0812 09:46:09.120058 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:46:09.120060 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:46:09.120105 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:46:09.120180 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:46:09.120761 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:46:09.121286 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:46:09.122064 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:46:09.122307 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:46:09.122408 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:46:09.125157 (+ 2749us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.99.2:39873: BlockingRecv error: recv got EOF from 127.13.99.2:39873 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 12 09:46:09 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.3@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 09:46:09 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.3@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 09:46:09.324577 28580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:46:09.318710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:40203 (local address 127.13.99.3:50479)
0812 09:46:09.319004 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0812 09:46:09.319019 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 09:46:09.319121 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:46:09.319358 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:46:09.319360 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:46:09.319558 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:46:09.319657 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:46:09.319663 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:46:09.320407 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:46:09.320411 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:46:09.321075 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:46:09.321086 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:46:09.321499 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:46:09.321502 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:46:09.321566 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:46:09.322139 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:46:09.322151 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:46:09.324378 (+ 2227us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.99.62:40203: Server oryx/127.13.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":118,"threads_started":1}
W20260812 09:46:09.324640 28578 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 09:46:09.318803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.3:50479 (local address 127.13.99.62:40203)
0812 09:46:09.318931 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0812 09:46:09.318940 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 09:46:09.319083 (+ 143us) server_negotiation.cc:407] Connection header received
0812 09:46:09.319199 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:46:09.319201 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:46:09.319247 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:46:09.319316 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:46:09.319753 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:46:09.320284 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:46:09.321190 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:46:09.321407 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:46:09.321487 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:46:09.324529 (+ 3042us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.99.3:50479: BlockingRecv error: recv got EOF from 127.13.99.3:50479 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Aug 12 09:46:09 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.1@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 09:46:09 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.1@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 09:46:09.944692 28582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:46:09.935253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:40203 (local address 127.13.99.1:44855)
0812 09:46:09.935547 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0812 09:46:09.935566 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 09:46:09.935693 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:46:09.935962 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:46:09.935964 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:46:09.936150 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:46:09.936245 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:46:09.936250 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:46:09.937071 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:46:09.937074 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:46:09.937628 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:46:09.937635 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:46:09.938096 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:46:09.938099 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:46:09.938159 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:46:09.942594 (+ 4435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:46:09.942612 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:46:09.944564 (+ 1952us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.99.62:40203: Server oryx/127.13.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":94,"threads_started":1}
W20260812 09:46:09.944818 28586 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 09:46:09.935300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.1:44855 (local address 127.13.99.62:40203)
0812 09:46:09.935635 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0812 09:46:09.935641 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 09:46:09.935659 (+ 18us) server_negotiation.cc:407] Connection header received
0812 09:46:09.935784 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:46:09.935787 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:46:09.935832 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:46:09.935929 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:46:09.936353 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:46:09.936966 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:46:09.937752 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:46:09.937985 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:46:09.938080 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:46:09.944685 (+ 6605us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.99.1:44855: BlockingRecv error: recv got EOF from 127.13.99.1:44855 (error 108)
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":64,"threads_started":1}
Aug 12 09:46:10 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.2@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 09:46:10 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.2@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 09:46:10.132110 28589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:46:10.125936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:40203 (local address 127.13.99.2:53521)
0812 09:46:10.126261 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0812 09:46:10.126280 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 09:46:10.126423 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:46:10.126736 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:46:10.126738 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:46:10.126953 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:46:10.127057 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:46:10.127063 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:46:10.127957 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:46:10.127960 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:46:10.128695 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:46:10.128707 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:46:10.129281 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:46:10.129284 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:46:10.129387 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:46:10.129963 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:46:10.129975 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:46:10.131980 (+ 2005us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.99.62:40203: Server oryx/127.13.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":124,"threads_started":1}
W20260812 09:46:10.132277 28586 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 09:46:10.126043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.2:53521 (local address 127.13.99.62:40203)
0812 09:46:10.126268 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0812 09:46:10.126273 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:46:10.126391 (+ 118us) server_negotiation.cc:407] Connection header received
0812 09:46:10.126572 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:46:10.126575 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:46:10.126626 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:46:10.126701 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:46:10.127184 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:46:10.127820 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:46:10.128793 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:46:10.129142 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:46:10.129258 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:46:10.132126 (+ 2868us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.99.2:53521: BlockingRecv error: recv got EOF from 127.13.99.2:53521 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Aug 12 09:46:10 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.3@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 09:46:10 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.3@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 09:46:10.331282 28590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:46:10.325524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:40203 (local address 127.13.99.3:45705)
0812 09:46:10.325816 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0812 09:46:10.325832 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 09:46:10.325949 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:46:10.326223 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:46:10.326226 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:46:10.326450 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:46:10.326563 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:46:10.326570 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:46:10.327467 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:46:10.327470 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:46:10.328223 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:46:10.328235 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:46:10.328708 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:46:10.328710 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:46:10.328774 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:46:10.329258 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:46:10.329269 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:46:10.331157 (+ 1888us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.99.62:40203: Server oryx/127.13.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":102,"threads_started":1}
W20260812 09:46:10.331420 28586 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 09:46:10.325590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.3:45705 (local address 127.13.99.62:40203)
0812 09:46:10.325767 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0812 09:46:10.325772 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:46:10.325977 (+ 205us) server_negotiation.cc:407] Connection header received
0812 09:46:10.326024 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:46:10.326026 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:46:10.326076 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:46:10.326152 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:46:10.326757 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:46:10.327359 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:46:10.328340 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:46:10.328619 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:46:10.328700 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:46:10.331297 (+ 2597us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.99.3:45705: BlockingRecv error: recv got EOF from 127.13.99.3:45705 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Aug 12 09:46:10 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.1@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 09:46:10 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.1@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 09:46:10.950979 28591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:46:10.945568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:40203 (local address 127.13.99.1:58259)
0812 09:46:10.945869 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0812 09:46:10.945882 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 09:46:10.945969 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:46:10.946234 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:46:10.946237 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:46:10.946423 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:46:10.946526 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:46:10.946531 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:46:10.947254 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:46:10.947257 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:46:10.947817 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:46:10.947825 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:46:10.948299 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:46:10.948303 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:46:10.948381 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:46:10.948923 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:46:10.948942 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:46:10.950858 (+ 1916us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.99.62:40203: Server oryx/127.13.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":84,"threads_started":1}
W20260812 09:46:10.951141 28592 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 09:46:10.945679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.1:58259 (local address 127.13.99.62:40203)
0812 09:46:10.945933 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0812 09:46:10.945937 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:46:10.945951 (+ 14us) server_negotiation.cc:407] Connection header received
0812 09:46:10.946068 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:46:10.946070 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:46:10.946115 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:46:10.946195 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:46:10.946629 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:46:10.947157 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:46:10.947922 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:46:10.948140 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:46:10.948217 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:46:10.951001 (+ 2784us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.99.1:58259: BlockingRecv error: recv got EOF from 127.13.99.1:58259 (error 108)
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":80,"threads_started":1}
Aug 12 09:46:11 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.2@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 09:46:11 dist-test-slave-jq4t krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.99.2@KRBTEST.COM for oryx/127.13.99.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 09:46:11.138968 28593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:46:11.132998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:40203 (local address 127.13.99.2:45429)
0812 09:46:11.133303 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0812 09:46:11.133316 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 09:46:11.133442 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:46:11.133724 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:46:11.133728 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:46:11.133963 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:46:11.134059 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:46:11.134066 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:46:11.134929 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:46:11.134933 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:46:11.135577 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:46:11.135587 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:46:11.136045 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:46:11.136049 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:46:11.136150 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:46:11.136759 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:46:11.136771 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:46:11.138842 (+ 2071us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.99.62:40203: Server oryx/127.13.99.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":87,"threads_started":1}
W20260812 09:46:11.139056 28592 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 09:46:11.133073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.2:45429 (local address 127.13.99.62:40203)
0812 09:46:11.133244 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0812 09:46:11.133248 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:46:11.133395 (+ 147us) server_negotiation.cc:407] Connection header received
0812 09:46:11.133547 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:46:11.133551 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:46:11.133607 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:46:11.133694 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:46:11.134194 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:46:11.134813 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:46:11.135695 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:46:11.135931 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:46:11.136041 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:46:11.138967 (+ 2926us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.99.2:45429: BlockingRecv error: recv got EOF from 127.13.99.2:45429 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260812 09:46:11.229635 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 27776
I20260812 09:46:11.249185 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 28037
I20260812 09:46:11.261516 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 28266
I20260812 09:46:11.284245 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 27645
2026-08-12T09:46:11Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18430 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-12T09:46:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T09:46:11Z Disabled control of system clock
I20260812 09:46:11.326913 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786527905902985-13708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786527905902985-13708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786527905902985-13708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.62:37821
--webserver_interface=127.13.99.62
--webserver_port=0
--builtin_ntp_servers=127.13.99.20:44405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.99.62:37821
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260812 09:46:11.442091 28598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:46:11.442368 28598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:46:11.442402 28598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:46:11.446720 28598 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 09:46:11.446794 28598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:46:11.446811 28598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:46:11.446828 28598 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 09:46:11.446844 28598 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 09:46:11.451808 28598 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:44405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786527905902985-13708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786527905902985-13708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.99.62:37821
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.62:37821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.99.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28598
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786527905902985-13708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:46:11.452950 28598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:46:11.454326 28598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:46:11.463303 28605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:46:11.463371 28611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:46:11.463353 28609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:46:11.463598 28598 server_base.cc:1034] running on GCE node
I20260812 09:46:11.464205 28598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:46:11.465428 28598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:46:11.466611 28598 hybrid_clock.cc:648] HybridClock initialized: now 1786527971466578 us; error 43 us; skew 500 ppm
I20260812 09:46:11.468716 28598 webserver.cc:460] Webserver started at http://127.13.99.62:35437/ using document root <none> and password file <none>
I20260812 09:46:11.469323 28598 fs_manager.cc:359] Metadata directory not provided
I20260812 09:46:11.469388 28598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:46:11.469661 28598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:46:11.471757 28598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786527905902985-13708-0/minicluster-data/master-0/data/instance:
uuid: "36ec4b492bc949848eb295bc0efa7d66"
format_stamp: "Formatted at 2026-08-12 09:46:11 on dist-test-slave-jq4t"
server_key: "7f30240bedbe8a871bdab716fe91ccc6"
server_key_iv: "a890e73d6e5a9c404d0c97178ea3f658"
server_key_version: "encryptionkey@0"
I20260812 09:46:11.472337 28598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786527905902985-13708-0/minicluster-data/master-0/wal/instance:
uuid: "36ec4b492bc949848eb295bc0efa7d66"
format_stamp: "Formatted at 2026-08-12 09:46:11 on dist-test-slave-jq4t"
server_key: "7f30240bedbe8a871bdab716fe91ccc6"
server_key_iv: "a890e73d6e5a9c404d0c97178ea3f658"
server_key_version: "encryptionkey@0"
I20260812 09:46:11.475986 28598 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260812 09:46:11.478421 28621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:46:11.479619 28598 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260812 09:46:11.479764 28598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786527905902985-13708-0/minicluster-data/master-0/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786527905902985-13708-0/minicluster-data/master-0/wal
uuid: "36ec4b492bc949848eb295bc0efa7d66"
format_stamp: "Formatted at 2026-08-12 09:46:11 on dist-test-slave-jq4t"
server_key: "7f30240bedbe8a871bdab716fe91ccc6"
server_key_iv: "a890e73d6e5a9c404d0c97178ea3f658"
server_key_version: "encryptionkey@0"
I20260812 09:46:11.479877 28598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786527905902985-13708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786527905902985-13708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786527905902985-13708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:46:11.491194 28598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:46:11.491806 28598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:46:11.491995 28598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:46:11.499068 28598 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.62:37821
I20260812 09:46:11.499073 28712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.62:37821 every 8 connection(s)
I20260812 09:46:11.499969 28598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
I20260812 09:46:11.502781 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 28598
I20260812 09:46:11.502907 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786527905902985-13708-0/minicluster-data/master-0/wal/instance
I20260812 09:46:11.503237 13708 external_mini_cluster.cc:1428] Setting key 551a0e21c794a0ad31f09d3cd4bbe6ec
I20260812 09:46:11.503379 28713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:46:11.509831 28713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36ec4b492bc949848eb295bc0efa7d66: Bootstrap starting.
I20260812 09:46:11.512091 28713 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 36ec4b492bc949848eb295bc0efa7d66: Neither blocks nor log segments found. Creating new log.
I20260812 09:46:11.512429 28717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:46:11.504687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60726 (local address 127.13.99.62:37821)
0812 09:46:11.505296 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0812 09:46:11.505311 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 09:46:11.505345 (+ 34us) server_negotiation.cc:407] Connection header received
0812 09:46:11.506152 (+ 807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:46:11.506172 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:46:11.506568 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:46:11.506893 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:46:11.507319 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:46:11.508284 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:46:11.509201 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:46:11.509667 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:46:11.509849 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:46:11.510746 (+ 897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:46:11.510751 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:46:11.510763 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 09:46:11.511146 (+ 383us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:46:11.511199 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 09:46:11.511219 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:46:11.511331 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0812 09:46:11.511456 (+ 125us) server_negotiation.cc:299] Negotiation successful
0812 09:46:11.511704 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":135,"threads_started":1}
I20260812 09:46:11.513047 28713 log.cc:824] T 00000000000000000000000000000000 P 36ec4b492bc949848eb295bc0efa7d66: Log is configured to *not* fsync() on all Append() calls
I20260812 09:46:11.514981 28713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36ec4b492bc949848eb295bc0efa7d66: No bootstrap required, opened a new log
I20260812 09:46:11.518491 28713 raft_consensus.cc:348] T 00000000000000000000000000000000 P 36ec4b492bc949848eb295bc0efa7d66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36ec4b492bc949848eb295bc0efa7d66" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 37821 } }
I20260812 09:46:11.518698 28713 raft_consensus.cc:374] T 00000000000000000000000000000000 P 36ec4b492bc949848eb295bc0efa7d66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:46:11.518771 28713 raft_consensus.cc:729] T 00000000000000000000000000000000 P 36ec4b492bc949848eb295bc0efa7d66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36ec4b492bc949848eb295bc0efa7d66, State: Initialized, Role: FOLLOWER
I20260812 09:46:11.519238 28713 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36ec4b492bc949848eb295bc0efa7d66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36ec4b492bc949848eb295bc0efa7d66" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 37821 } }
I20260812 09:46:11.519429 28713 raft_consensus.cc:388] T 00000000000000000000000000000000 P 36ec4b492bc949848eb295bc0efa7d66 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 09:46:11.519522 28713 raft_consensus.cc:482] T 00000000000000000000000000000000 P 36ec4b492bc949848eb295bc0efa7d66 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 09:46:11.519621 28713 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 36ec4b492bc949848eb295bc0efa7d66 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:46:11.520555 28713 raft_consensus.cc:504] T 00000000000000000000000000000000 P 36ec4b492bc949848eb295bc0efa7d66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36ec4b492bc949848eb295bc0efa7d66" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 37821 } }
I20260812 09:46:11.520851 28713 leader_election.cc:302] T 00000000000000000000000000000000 P 36ec4b492bc949848eb295bc0efa7d66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 36ec4b492bc949848eb295bc0efa7d66; no voters:
I20260812 09:46:11.521102 28713 leader_election.cc:288] T 00000000000000000000000000000000 P 36ec4b492bc949848eb295bc0efa7d66 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 09:46:11.521245 28721 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 36ec4b492bc949848eb295bc0efa7d66 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 09:46:11.521514 28721 raft_consensus.cc:686] T 00000000000000000000000000000000 P 36ec4b492bc949848eb295bc0efa7d66 [term 1 LEADER]: Becoming Leader. State: Replica: 36ec4b492bc949848eb295bc0efa7d66, State: Running, Role: LEADER
I20260812 09:46:11.521831 28721 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36ec4b492bc949848eb295bc0efa7d66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36ec4b492bc949848eb295bc0efa7d66" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 37821 } }
I20260812 09:46:11.522356 28713 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36ec4b492bc949848eb295bc0efa7d66 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 09:46:11.523591 28724 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36ec4b492bc949848eb295bc0efa7d66 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36ec4b492bc949848eb295bc0efa7d66. Latest consensus state: current_term: 1 leader_uuid: "36ec4b492bc949848eb295bc0efa7d66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36ec4b492bc949848eb295bc0efa7d66" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 37821 } } }
I20260812 09:46:11.523620 28723 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36ec4b492bc949848eb295bc0efa7d66 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "36ec4b492bc949848eb295bc0efa7d66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36ec4b492bc949848eb295bc0efa7d66" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 37821 } } }
I20260812 09:46:11.523708 28724 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36ec4b492bc949848eb295bc0efa7d66 [sys.catalog]: This master's current role is: LEADER
I20260812 09:46:11.523729 28723 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36ec4b492bc949848eb295bc0efa7d66 [sys.catalog]: This master's current role is: LEADER
I20260812 09:46:11.524199 28732 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 09:46:11.527088 28732 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 09:46:11.532979 28732 catalog_manager.cc:1347] Generated new cluster ID: 6cfde66d45c046d0b029921c13a4c0d5
I20260812 09:46:11.533092 28732 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 09:46:11.543920 28732 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 09:46:11.544842 28732 catalog_manager.cc:1504] Loading token signing keys...
I20260812 09:46:11.557489 28732 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 36ec4b492bc949848eb295bc0efa7d66: Generated new TSK 0
I20260812 09:46:11.558115 28732 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 09:46:11.561069 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.99.1:0
--local_ip_for_outbound_sockets=127.13.99.1
--webserver_interface=127.13.99.1
--webserver_port=0
--tserver_master_addrs=127.13.99.62:37821
--builtin_ntp_servers=127.13.99.20:44405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260812 09:46:11.690315 28755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:46:11.690554 28755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:46:11.690589 28755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:46:11.694598 28755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:46:11.694648 28755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:46:11.694698 28755 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.99.1
I20260812 09:46:11.699412 28755 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:44405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.99.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.99.1
--webserver_port=0
--tserver_master_addrs=127.13.99.62:37821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28755
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.99.1
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:46:11.700426 28755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:46:11.701778 28755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 09:46:11.711454 28755 server_base.cc:1034] running on GCE node
W20260812 09:46:11.711412 28760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:46:11.711476 28764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:46:11.711704 28761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:46:11.712152 28755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:46:11.712766 28755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:46:11.713958 28755 hybrid_clock.cc:648] HybridClock initialized: now 1786527971713936 us; error 32 us; skew 500 ppm
I20260812 09:46:11.715819 28755 webserver.cc:460] Webserver started at http://127.13.99.1:36487/ using document root <none> and password file <none>
I20260812 09:46:11.716398 28755 fs_manager.cc:359] Metadata directory not provided
I20260812 09:46:11.716511 28755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:46:11.716753 28755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:46:11.718763 28755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-0/data/instance:
uuid: "b44548d3ad884456b451e3b07a765b45"
format_stamp: "Formatted at 2026-08-12 09:46:11 on dist-test-slave-jq4t"
server_key: "65090caa8617986c41bda09f8b355f21"
server_key_iv: "02a706f8f10d1af2beaa3d2b0e1e5a11"
server_key_version: "encryptionkey@0"
I20260812 09:46:11.719332 28755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-0/wal/instance:
uuid: "b44548d3ad884456b451e3b07a765b45"
format_stamp: "Formatted at 2026-08-12 09:46:11 on dist-test-slave-jq4t"
server_key: "65090caa8617986c41bda09f8b355f21"
server_key_iv: "02a706f8f10d1af2beaa3d2b0e1e5a11"
server_key_version: "encryptionkey@0"
I20260812 09:46:11.723063 28755 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260812 09:46:11.725680 28771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:46:11.726905 28755 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260812 09:46:11.727048 28755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-0/wal
uuid: "b44548d3ad884456b451e3b07a765b45"
format_stamp: "Formatted at 2026-08-12 09:46:11 on dist-test-slave-jq4t"
server_key: "65090caa8617986c41bda09f8b355f21"
server_key_iv: "02a706f8f10d1af2beaa3d2b0e1e5a11"
server_key_version: "encryptionkey@0"
I20260812 09:46:11.727160 28755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:46:11.742553 28755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:46:11.743216 28755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:46:11.743413 28755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:46:11.743990 28755 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 09:46:11.745031 28755 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:46:11.745103 28755 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 09:46:11.745174 28755 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:46:11.745216 28755 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 09:46:11.753938 28755 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.1:33189
I20260812 09:46:11.754079 28970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.1:33189 every 8 connection(s)
I20260812 09:46:11.755091 28755 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-0/data/info.pb
I20260812 09:46:11.759603 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 28755
I20260812 09:46:11.759759 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-0/wal/instance
I20260812 09:46:11.760145 13708 external_mini_cluster.cc:1428] Setting key 4f232680ac3db2466b978ab5a11f750b
I20260812 09:46:11.762394 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.99.2:0
--local_ip_for_outbound_sockets=127.13.99.2
--webserver_interface=127.13.99.2
--webserver_port=0
--tserver_master_addrs=127.13.99.62:37821
--builtin_ntp_servers=127.13.99.20:44405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260812 09:46:11.766667 28717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:46:11.757112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.1:33597 (local address 127.13.99.62:37821)
0812 09:46:11.757273 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0812 09:46:11.757277 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:46:11.758396 (+ 1119us) server_negotiation.cc:407] Connection header received
0812 09:46:11.760297 (+ 1901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:46:11.760303 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:46:11.760368 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:46:11.760452 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:46:11.761655 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:46:11.762242 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:46:11.764257 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:46:11.764702 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:46:11.764829 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:46:11.766040 (+ 1211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:46:11.766047 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:46:11.766052 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 09:46:11.766071 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:46:11.766098 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 09:46:11.766111 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:46:11.766193 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0812 09:46:11.766352 (+ 159us) server_negotiation.cc:299] Negotiation successful
0812 09:46:11.766448 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 09:46:11.767493 28975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:46:11.757533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:37821 (local address 127.13.99.1:33597)
0812 09:46:11.758241 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0812 09:46:11.758285 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 09:46:11.759973 (+ 1688us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:46:11.760719 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:46:11.760727 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:46:11.760845 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:46:11.761494 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:46:11.761516 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:46:11.762400 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:46:11.762406 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:46:11.764085 (+ 1679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:46:11.764107 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:46:11.764887 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:46:11.764891 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:46:11.765052 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:46:11.765746 (+ 694us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 09:46:11.765747 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:46:11.765789 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 09:46:11.765875 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:46:11.766200 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:46:11.766208 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:46:11.766213 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 09:46:11.766315 (+ 102us) client_negotiation.cc:240] Negotiation successful
0812 09:46:11.766560 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":113,"threads_started":1}
I20260812 09:46:11.768981 28973 heartbeater.cc:347] Connected to a master server at 127.13.99.62:37821
I20260812 09:46:11.769253 28973 heartbeater.cc:464] Registering TS with master...
I20260812 09:46:11.769796 28973 heartbeater.cc:511] Master 127.13.99.62:37821 requested a full tablet report, sending...
I20260812 09:46:11.771605 28649 ts_manager.cc:194] Registered new tserver with Master: b44548d3ad884456b451e3b07a765b45 (127.13.99.1:33189)
I20260812 09:46:11.773440 28649 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.99.1:33597
W20260812 09:46:11.880671 28977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:46:11.880914 28977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:46:11.880957 28977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:46:11.884971 28977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:46:11.885021 28977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:46:11.885074 28977 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.99.2
I20260812 09:46:11.889611 28977 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:44405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.99.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.99.2
--webserver_port=0
--tserver_master_addrs=127.13.99.62:37821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.99.2
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:46:11.890931 28977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:46:11.892338 28977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:46:11.901917 28992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:46:11.901988 28986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:46:11.901960 28977 server_base.cc:1034] running on GCE node
W20260812 09:46:11.901949 28987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:46:11.902511 28977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:46:11.903157 28977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:46:11.904327 28977 hybrid_clock.cc:648] HybridClock initialized: now 1786527971904310 us; error 27 us; skew 500 ppm
I20260812 09:46:11.906628 28977 webserver.cc:460] Webserver started at http://127.13.99.2:36591/ using document root <none> and password file <none>
I20260812 09:46:11.907209 28977 fs_manager.cc:359] Metadata directory not provided
I20260812 09:46:11.907305 28977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:46:11.907521 28977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:46:11.909596 28977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-1/data/instance:
uuid: "2ee805f17d1d417fbd3a45621793c836"
format_stamp: "Formatted at 2026-08-12 09:46:11 on dist-test-slave-jq4t"
server_key: "e54efd90a79d5146f0c0624660a44d72"
server_key_iv: "efea630c4aa69aac68558b39cec84f8b"
server_key_version: "encryptionkey@0"
I20260812 09:46:11.910181 28977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-1/wal/instance:
uuid: "2ee805f17d1d417fbd3a45621793c836"
format_stamp: "Formatted at 2026-08-12 09:46:11 on dist-test-slave-jq4t"
server_key: "e54efd90a79d5146f0c0624660a44d72"
server_key_iv: "efea630c4aa69aac68558b39cec84f8b"
server_key_version: "encryptionkey@0"
I20260812 09:46:11.913786 28977 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 09:46:11.916292 28998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:46:11.917558 28977 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 09:46:11.917703 28977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-1/wal
uuid: "2ee805f17d1d417fbd3a45621793c836"
format_stamp: "Formatted at 2026-08-12 09:46:11 on dist-test-slave-jq4t"
server_key: "e54efd90a79d5146f0c0624660a44d72"
server_key_iv: "efea630c4aa69aac68558b39cec84f8b"
server_key_version: "encryptionkey@0"
I20260812 09:46:11.917816 28977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:46:11.933462 28977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:46:11.934105 28977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:46:11.934294 28977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:46:11.934899 28977 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 09:46:11.935858 28977 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:46:11.935938 28977 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 09:46:11.936009 28977 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:46:11.936062 28977 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 09:46:11.945164 28977 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.2:46867
I20260812 09:46:11.945188 29187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.2:46867 every 8 connection(s)
I20260812 09:46:11.946128 28977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-1/data/info.pb
I20260812 09:46:11.951287 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 28977
I20260812 09:46:11.951421 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-1/wal/instance
I20260812 09:46:11.951756 13708 external_mini_cluster.cc:1428] Setting key cf64d7ba8db77b6cdaea486c4a8e6758
I20260812 09:46:11.953918 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.99.3:0
--local_ip_for_outbound_sockets=127.13.99.3
--webserver_interface=127.13.99.3
--webserver_port=0
--tserver_master_addrs=127.13.99.62:37821
--builtin_ntp_servers=127.13.99.20:44405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260812 09:46:11.956735 28717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:46:11.948008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.2:59801 (local address 127.13.99.62:37821)
0812 09:46:11.948154 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0812 09:46:11.948158 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:46:11.949313 (+ 1155us) server_negotiation.cc:407] Connection header received
0812 09:46:11.950683 (+ 1370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:46:11.950686 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:46:11.950743 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:46:11.950848 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:46:11.952122 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:46:11.953032 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:46:11.954505 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:46:11.954758 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:46:11.954894 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:46:11.956200 (+ 1306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:46:11.956203 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:46:11.956205 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 09:46:11.956217 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:46:11.956237 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 09:46:11.956246 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:46:11.956315 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0812 09:46:11.956569 (+ 254us) server_negotiation.cc:299] Negotiation successful
0812 09:46:11.956624 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260812 09:46:11.957952 29192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:46:11.948379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:37821 (local address 127.13.99.2:59801)
0812 09:46:11.949156 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0812 09:46:11.949197 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0812 09:46:11.950456 (+ 1259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:46:11.951072 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:46:11.951083 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:46:11.951194 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:46:11.951943 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:46:11.951965 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:46:11.953184 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:46:11.953193 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:46:11.954362 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:46:11.954374 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:46:11.954934 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:46:11.954939 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:46:11.955123 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:46:11.955839 (+ 716us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 09:46:11.955843 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:46:11.955902 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 09:46:11.955993 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:46:11.956341 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:46:11.956353 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:46:11.956360 (+ 7us) client_negotiation.cc:769] Sending connection context
0812 09:46:11.956549 (+ 189us) client_negotiation.cc:240] Negotiation successful
0812 09:46:11.956869 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":131,"threads_started":1}
I20260812 09:46:11.959242 29189 heartbeater.cc:347] Connected to a master server at 127.13.99.62:37821
I20260812 09:46:11.959507 29189 heartbeater.cc:464] Registering TS with master...
I20260812 09:46:11.960053 29189 heartbeater.cc:511] Master 127.13.99.62:37821 requested a full tablet report, sending...
I20260812 09:46:11.961212 28649 ts_manager.cc:194] Registered new tserver with Master: 2ee805f17d1d417fbd3a45621793c836 (127.13.99.2:46867)
I20260812 09:46:11.961783 28649 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.99.2:59801
W20260812 09:46:12.071319 29193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:46:12.071645 29193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:46:12.071724 29193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:46:12.075914 29193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:46:12.076007 29193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:46:12.076097 29193 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.99.3
I20260812 09:46:12.081045 29193 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:44405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.99.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.99.3
--webserver_port=0
--tserver_master_addrs=127.13.99.62:37821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29193
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.99.3
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:46:12.082177 29193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:46:12.083595 29193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 09:46:12.094022 29193 server_base.cc:1034] running on GCE node
W20260812 09:46:12.093920 29203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:46:12.093920 29208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:46:12.093917 29201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:46:12.094637 29193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:46:12.095240 29193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:46:12.096436 29193 hybrid_clock.cc:648] HybridClock initialized: now 1786527972096411 us; error 36 us; skew 500 ppm
I20260812 09:46:12.098683 29193 webserver.cc:460] Webserver started at http://127.13.99.3:45929/ using document root <none> and password file <none>
I20260812 09:46:12.099300 29193 fs_manager.cc:359] Metadata directory not provided
I20260812 09:46:12.099398 29193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:46:12.099638 29193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:46:12.101831 29193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-2/data/instance:
uuid: "2995780609884fbd8a2d9aee4e02a7cd"
format_stamp: "Formatted at 2026-08-12 09:46:12 on dist-test-slave-jq4t"
server_key: "a6261458440910493bab5389b4057287"
server_key_iv: "3bffb295f51e3cf3e8fde3287013ed37"
server_key_version: "encryptionkey@0"
I20260812 09:46:12.102416 29193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-2/wal/instance:
uuid: "2995780609884fbd8a2d9aee4e02a7cd"
format_stamp: "Formatted at 2026-08-12 09:46:12 on dist-test-slave-jq4t"
server_key: "a6261458440910493bab5389b4057287"
server_key_iv: "3bffb295f51e3cf3e8fde3287013ed37"
server_key_version: "encryptionkey@0"
I20260812 09:46:12.106217 29193 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260812 09:46:12.108816 29218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:46:12.110025 29193 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260812 09:46:12.110157 29193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-2/wal
uuid: "2995780609884fbd8a2d9aee4e02a7cd"
format_stamp: "Formatted at 2026-08-12 09:46:12 on dist-test-slave-jq4t"
server_key: "a6261458440910493bab5389b4057287"
server_key_iv: "3bffb295f51e3cf3e8fde3287013ed37"
server_key_version: "encryptionkey@0"
I20260812 09:46:12.110267 29193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:46:12.134013 29193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:46:12.134711 29193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:46:12.134912 29193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:46:12.135519 29193 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 09:46:12.136471 29193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:46:12.136560 29193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 09:46:12.136633 29193 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:46:12.136677 29193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 09:46:12.145820 29193 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.3:35983
I20260812 09:46:12.145884 29392 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.3:35983 every 8 connection(s)
I20260812 09:46:12.146763 29193 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-2/data/info.pb
I20260812 09:46:12.152532 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 29193
I20260812 09:46:12.152653 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-2/wal/instance
I20260812 09:46:12.152962 13708 external_mini_cluster.cc:1428] Setting key 8c0c3e726e233a63118179a39e2f58ad
I20260812 09:46:12.156723 28717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:46:12.148686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.3:48567 (local address 127.13.99.62:37821)
0812 09:46:12.148848 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0812 09:46:12.148853 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:46:12.149950 (+ 1097us) server_negotiation.cc:407] Connection header received
0812 09:46:12.151426 (+ 1476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:46:12.151428 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:46:12.151481 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:46:12.151557 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:46:12.152523 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:46:12.153330 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:46:12.154828 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:46:12.155093 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:46:12.155214 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:46:12.156278 (+ 1064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:46:12.156281 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:46:12.156283 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 09:46:12.156295 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:46:12.156315 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 09:46:12.156322 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:46:12.156381 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0812 09:46:12.156556 (+ 175us) server_negotiation.cc:299] Negotiation successful
0812 09:46:12.156611 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260812 09:46:12.157636 29397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:46:12.149121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:37821 (local address 127.13.99.3:48567)
0812 09:46:12.149798 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0812 09:46:12.149842 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 09:46:12.151217 (+ 1375us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:46:12.151739 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:46:12.151746 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:46:12.151835 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:46:12.152349 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:46:12.152363 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:46:12.153466 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:46:12.153474 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:46:12.154707 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:46:12.154718 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:46:12.155275 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:46:12.155278 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:46:12.155437 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:46:12.156010 (+ 573us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 09:46:12.156012 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:46:12.156066 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 09:46:12.156135 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:46:12.156382 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:46:12.156387 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:46:12.156391 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 09:46:12.156500 (+ 109us) client_negotiation.cc:240] Negotiation successful
0812 09:46:12.156802 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":110,"threads_started":1}
I20260812 09:46:12.158891 29393 heartbeater.cc:347] Connected to a master server at 127.13.99.62:37821
I20260812 09:46:12.159163 29393 heartbeater.cc:464] Registering TS with master...
I20260812 09:46:12.159708 29393 heartbeater.cc:511] Master 127.13.99.62:37821 requested a full tablet report, sending...
I20260812 09:46:12.160804 28649 ts_manager.cc:194] Registered new tserver with Master: 2995780609884fbd8a2d9aee4e02a7cd (127.13.99.3:35983)
I20260812 09:46:12.161381 28649 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.99.3:48567
I20260812 09:46:12.167098 13708 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260812 09:46:12.170033 28717 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 09:46:12.169143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60742 (local address 127.13.99.62:37821)
0812 09:46:12.169264 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0812 09:46:12.169268 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:46:12.169452 (+ 184us) server_negotiation.cc:407] Connection header received
0812 09:46:12.169553 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:46:12.169555 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:46:12.169594 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:46:12.169665 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:46:12.169888 (+ 223us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60742: BlockingRecv error: recv got EOF from 127.0.0.1:60742 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260812 09:46:12.170338 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 28755
I20260812 09:46:12.190187 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 28977
I20260812 09:46:12.210419 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 29193
I20260812 09:46:12.232169 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 28598
2026-08-12T09:46:12Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (942 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-12T09:46:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T09:46:12Z Disabled control of system clock
I20260812 09:46:12.269884 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786527905902985-13708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786527905902985-13708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786527905902985-13708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.62:37945
--webserver_interface=127.13.99.62
--webserver_port=0
--builtin_ntp_servers=127.13.99.20:35733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.99.62:37945
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260812 09:46:12.381635 29412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:46:12.381904 29412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:46:12.381940 29412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:46:12.385987 29412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 09:46:12.386047 29412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:46:12.386062 29412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:46:12.386080 29412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 09:46:12.386096 29412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 09:46:12.390867 29412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:35733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786527905902985-13708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786527905902985-13708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.99.62:37945
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.62:37945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.99.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786527905902985-13708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:46:12.392024 29412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:46:12.393496 29412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:46:12.402133 29428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:46:12.402170 29421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:46:12.402166 29420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:46:12.402547 29412 server_base.cc:1034] running on GCE node
I20260812 09:46:12.403126 29412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:46:12.404183 29412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:46:12.405352 29412 hybrid_clock.cc:648] HybridClock initialized: now 1786527972405339 us; error 27 us; skew 500 ppm
I20260812 09:46:12.407222 29412 webserver.cc:460] Webserver started at http://127.13.99.62:39833/ using document root <none> and password file <none>
I20260812 09:46:12.407756 29412 fs_manager.cc:359] Metadata directory not provided
I20260812 09:46:12.407815 29412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:46:12.407994 29412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:46:12.410075 29412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786527905902985-13708-0/minicluster-data/master-0/data/instance:
uuid: "ee285d38f89149c4857e619a26e66e1a"
format_stamp: "Formatted at 2026-08-12 09:46:12 on dist-test-slave-jq4t"
server_key: "96268f3b2998c32bd4f17f9ecf1d3b5a"
server_key_iv: "d7cc8c9394fee6aaea884f1a93992613"
server_key_version: "encryptionkey@0"
I20260812 09:46:12.410593 29412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786527905902985-13708-0/minicluster-data/master-0/wal/instance:
uuid: "ee285d38f89149c4857e619a26e66e1a"
format_stamp: "Formatted at 2026-08-12 09:46:12 on dist-test-slave-jq4t"
server_key: "96268f3b2998c32bd4f17f9ecf1d3b5a"
server_key_iv: "d7cc8c9394fee6aaea884f1a93992613"
server_key_version: "encryptionkey@0"
I20260812 09:46:12.414352 29412 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260812 09:46:12.416986 29438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:46:12.418272 29412 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 09:46:12.418395 29412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786527905902985-13708-0/minicluster-data/master-0/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786527905902985-13708-0/minicluster-data/master-0/wal
uuid: "ee285d38f89149c4857e619a26e66e1a"
format_stamp: "Formatted at 2026-08-12 09:46:12 on dist-test-slave-jq4t"
server_key: "96268f3b2998c32bd4f17f9ecf1d3b5a"
server_key_iv: "d7cc8c9394fee6aaea884f1a93992613"
server_key_version: "encryptionkey@0"
I20260812 09:46:12.418488 29412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786527905902985-13708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786527905902985-13708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786527905902985-13708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:46:12.423084 29412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:46:12.423667 29412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:46:12.423821 29412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:46:12.430902 29412 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.62:37945
I20260812 09:46:12.430909 29523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.62:37945 every 8 connection(s)
I20260812 09:46:12.431852 29412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
I20260812 09:46:12.435290 29526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:46:12.435598 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 29412
I20260812 09:46:12.435722 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786527905902985-13708-0/minicluster-data/master-0/wal/instance
I20260812 09:46:12.436043 13708 external_mini_cluster.cc:1428] Setting key bc0ca51103b2e901fedb55b4e5371170
I20260812 09:46:12.441799 29534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:46:12.437577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50002 (local address 127.13.99.62:37945)
0812 09:46:12.438238 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0812 09:46:12.438253 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 09:46:12.438286 (+ 33us) server_negotiation.cc:407] Connection header received
0812 09:46:12.438966 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:46:12.438983 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:46:12.439316 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:46:12.439588 (+ 272us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:46:12.440464 (+ 876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:46:12.440468 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:46:12.440525 (+ 57us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 09:46:12.440542 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:46:12.440600 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 09:46:12.440617 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:46:12.440733 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0812 09:46:12.440875 (+ 142us) server_negotiation.cc:299] Negotiation successful
0812 09:46:12.441158 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":155,"threads_started":1}
I20260812 09:46:12.441963 29526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee285d38f89149c4857e619a26e66e1a: Bootstrap starting.
I20260812 09:46:12.444339 29526 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee285d38f89149c4857e619a26e66e1a: Neither blocks nor log segments found. Creating new log.
I20260812 09:46:12.445081 29526 log.cc:824] T 00000000000000000000000000000000 P ee285d38f89149c4857e619a26e66e1a: Log is configured to *not* fsync() on all Append() calls
I20260812 09:46:12.446991 29526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee285d38f89149c4857e619a26e66e1a: No bootstrap required, opened a new log
I20260812 09:46:12.450389 29526 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee285d38f89149c4857e619a26e66e1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee285d38f89149c4857e619a26e66e1a" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 37945 } }
I20260812 09:46:12.450601 29526 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee285d38f89149c4857e619a26e66e1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:46:12.450688 29526 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee285d38f89149c4857e619a26e66e1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee285d38f89149c4857e619a26e66e1a, State: Initialized, Role: FOLLOWER
I20260812 09:46:12.451107 29526 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee285d38f89149c4857e619a26e66e1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee285d38f89149c4857e619a26e66e1a" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 37945 } }
I20260812 09:46:12.451253 29526 raft_consensus.cc:388] T 00000000000000000000000000000000 P ee285d38f89149c4857e619a26e66e1a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 09:46:12.451316 29526 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee285d38f89149c4857e619a26e66e1a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 09:46:12.451447 29526 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee285d38f89149c4857e619a26e66e1a [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:46:12.452272 29526 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee285d38f89149c4857e619a26e66e1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee285d38f89149c4857e619a26e66e1a" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 37945 } }
I20260812 09:46:12.452643 29526 leader_election.cc:302] T 00000000000000000000000000000000 P ee285d38f89149c4857e619a26e66e1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee285d38f89149c4857e619a26e66e1a; no voters:
I20260812 09:46:12.452941 29526 leader_election.cc:288] T 00000000000000000000000000000000 P ee285d38f89149c4857e619a26e66e1a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 09:46:12.453095 29536 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee285d38f89149c4857e619a26e66e1a [term 1 FOLLOWER]: Leader election won for term 1
I20260812 09:46:12.453356 29536 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee285d38f89149c4857e619a26e66e1a [term 1 LEADER]: Becoming Leader. State: Replica: ee285d38f89149c4857e619a26e66e1a, State: Running, Role: LEADER
I20260812 09:46:12.453702 29536 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee285d38f89149c4857e619a26e66e1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee285d38f89149c4857e619a26e66e1a" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 37945 } }
I20260812 09:46:12.454170 29526 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee285d38f89149c4857e619a26e66e1a [sys.catalog]: configured and running, proceeding with master startup.
I20260812 09:46:12.455385 29537 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee285d38f89149c4857e619a26e66e1a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ee285d38f89149c4857e619a26e66e1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee285d38f89149c4857e619a26e66e1a" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 37945 } } }
I20260812 09:46:12.455525 29537 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee285d38f89149c4857e619a26e66e1a [sys.catalog]: This master's current role is: LEADER
I20260812 09:46:12.455749 29538 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee285d38f89149c4857e619a26e66e1a [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee285d38f89149c4857e619a26e66e1a. Latest consensus state: current_term: 1 leader_uuid: "ee285d38f89149c4857e619a26e66e1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee285d38f89149c4857e619a26e66e1a" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 37945 } } }
I20260812 09:46:12.455910 29538 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee285d38f89149c4857e619a26e66e1a [sys.catalog]: This master's current role is: LEADER
I20260812 09:46:12.455920 29546 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 09:46:12.459293 29546 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 09:46:12.464588 29546 catalog_manager.cc:1347] Generated new cluster ID: eb3fa812579244dca777940ca586ae20
I20260812 09:46:12.464660 29546 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 09:46:12.473268 29546 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 09:46:12.473578 29546 catalog_manager.cc:1504] Loading token signing keys...
I20260812 09:46:12.485626 29546 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ee285d38f89149c4857e619a26e66e1a: Generated new TSK 0
I20260812 09:46:12.486404 29546 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 09:46:12.491669 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.99.1:0
--local_ip_for_outbound_sockets=127.13.99.1
--webserver_interface=127.13.99.1
--webserver_port=0
--tserver_master_addrs=127.13.99.62:37945
--builtin_ntp_servers=127.13.99.20:35733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260812 09:46:12.607609 29562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:46:12.607925 29562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:46:12.607966 29562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:46:12.611901 29562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:46:12.611951 29562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:46:12.612005 29562 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.99.1
I20260812 09:46:12.616601 29562 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:35733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.99.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.99.1
--webserver_port=0
--tserver_master_addrs=127.13.99.62:37945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29562
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.99.1
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:46:12.617666 29562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:46:12.619122 29562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:46:12.629010 29570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:46:12.629010 29569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:46:12.629073 29562 server_base.cc:1034] running on GCE node
W20260812 09:46:12.629261 29574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:46:12.629667 29562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:46:12.630308 29562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:46:12.631510 29562 hybrid_clock.cc:648] HybridClock initialized: now 1786527972631492 us; error 27 us; skew 500 ppm
I20260812 09:46:12.633378 29562 webserver.cc:460] Webserver started at http://127.13.99.1:40821/ using document root <none> and password file <none>
I20260812 09:46:12.633960 29562 fs_manager.cc:359] Metadata directory not provided
I20260812 09:46:12.634055 29562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:46:12.634292 29562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:46:12.636296 29562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-0/data/instance:
uuid: "0f830c0583b0489f84012d2f20fe24fc"
format_stamp: "Formatted at 2026-08-12 09:46:12 on dist-test-slave-jq4t"
server_key: "e231d15c64b20f82af3ec595f08c7ca0"
server_key_iv: "0ef7ca97a2ebfca751ecd03641121e23"
server_key_version: "encryptionkey@0"
I20260812 09:46:12.636917 29562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-0/wal/instance:
uuid: "0f830c0583b0489f84012d2f20fe24fc"
format_stamp: "Formatted at 2026-08-12 09:46:12 on dist-test-slave-jq4t"
server_key: "e231d15c64b20f82af3ec595f08c7ca0"
server_key_iv: "0ef7ca97a2ebfca751ecd03641121e23"
server_key_version: "encryptionkey@0"
I20260812 09:46:12.640970 29562 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260812 09:46:12.643359 29588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:46:12.644580 29562 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 09:46:12.644704 29562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-0/wal
uuid: "0f830c0583b0489f84012d2f20fe24fc"
format_stamp: "Formatted at 2026-08-12 09:46:12 on dist-test-slave-jq4t"
server_key: "e231d15c64b20f82af3ec595f08c7ca0"
server_key_iv: "0ef7ca97a2ebfca751ecd03641121e23"
server_key_version: "encryptionkey@0"
I20260812 09:46:12.644811 29562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:46:12.649204 29562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:46:12.649789 29562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:46:12.649976 29562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:46:12.650513 29562 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 09:46:12.651405 29562 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:46:12.651481 29562 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 09:46:12.651552 29562 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:46:12.651604 29562 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 09:46:12.660673 29562 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.1:43965
I20260812 09:46:12.660735 29777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.1:43965 every 8 connection(s)
I20260812 09:46:12.661578 29562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-0/data/info.pb
I20260812 09:46:12.667706 29534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:46:12.663465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.1:33847 (local address 127.13.99.62:37945)
0812 09:46:12.663633 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0812 09:46:12.663638 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:46:12.664712 (+ 1074us) server_negotiation.cc:407] Connection header received
0812 09:46:12.666040 (+ 1328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:46:12.666043 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:46:12.666084 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:46:12.666163 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:46:12.667300 (+ 1137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:46:12.667303 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:46:12.667306 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 09:46:12.667320 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:46:12.667341 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 09:46:12.667350 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:46:12.667429 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0812 09:46:12.667561 (+ 132us) server_negotiation.cc:299] Negotiation successful
0812 09:46:12.667600 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 09:46:12.668623 29781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:46:12.663824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:37945 (local address 127.13.99.1:33847)
0812 09:46:12.664527 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0812 09:46:12.664590 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0812 09:46:12.665806 (+ 1216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:46:12.666328 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:46:12.666335 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:46:12.666461 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:46:12.667039 (+ 578us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 09:46:12.667041 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:46:12.667080 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 09:46:12.667146 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:46:12.667431 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:46:12.667438 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:46:12.667442 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 09:46:12.667574 (+ 132us) client_negotiation.cc:240] Negotiation successful
0812 09:46:12.667787 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":108,"threads_started":1}
I20260812 09:46:12.668823 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 29562
I20260812 09:46:12.668951 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-0/wal/instance
I20260812 09:46:12.669313 13708 external_mini_cluster.cc:1428] Setting key c81bfb764e9825a88514efbfdaa6568a
I20260812 09:46:12.669974 29778 heartbeater.cc:347] Connected to a master server at 127.13.99.62:37945
I20260812 09:46:12.670214 29778 heartbeater.cc:464] Registering TS with master...
I20260812 09:46:12.670666 29778 heartbeater.cc:511] Master 127.13.99.62:37945 requested a full tablet report, sending...
I20260812 09:46:12.671603 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.99.2:0
--local_ip_for_outbound_sockets=127.13.99.2
--webserver_interface=127.13.99.2
--webserver_port=0
--tserver_master_addrs=127.13.99.62:37945
--builtin_ntp_servers=127.13.99.20:35733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260812 09:46:12.672359 29460 ts_manager.cc:194] Registered new tserver with Master: 0f830c0583b0489f84012d2f20fe24fc (127.13.99.1:43965)
W20260812 09:46:12.785943 29783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:46:12.786186 29783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:46:12.786224 29783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:46:12.790307 29783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:46:12.790359 29783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:46:12.790412 29783 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.99.2
I20260812 09:46:12.795012 29783 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:35733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.99.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.99.2
--webserver_port=0
--tserver_master_addrs=127.13.99.62:37945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29783
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.99.2
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:46:12.796051 29783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:46:12.797444 29783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:46:12.807128 29793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:46:12.807255 29783 server_base.cc:1034] running on GCE node
W20260812 09:46:12.807133 29798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:46:12.807147 29794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:46:12.807924 29783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:46:12.808651 29783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:46:12.809836 29783 hybrid_clock.cc:648] HybridClock initialized: now 1786527972809819 us; error 28 us; skew 500 ppm
I20260812 09:46:12.811663 29783 webserver.cc:460] Webserver started at http://127.13.99.2:34985/ using document root <none> and password file <none>
I20260812 09:46:12.812269 29783 fs_manager.cc:359] Metadata directory not provided
I20260812 09:46:12.812340 29783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:46:12.812639 29783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:46:12.814774 29783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-1/data/instance:
uuid: "3412b9c64bf3436db2fca81e7c9bdc85"
format_stamp: "Formatted at 2026-08-12 09:46:12 on dist-test-slave-jq4t"
server_key: "8614204776aed09b0119ebc3186946f7"
server_key_iv: "625ba1c8d61924826e20e90be4316f00"
server_key_version: "encryptionkey@0"
I20260812 09:46:12.815366 29783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-1/wal/instance:
uuid: "3412b9c64bf3436db2fca81e7c9bdc85"
format_stamp: "Formatted at 2026-08-12 09:46:12 on dist-test-slave-jq4t"
server_key: "8614204776aed09b0119ebc3186946f7"
server_key_iv: "625ba1c8d61924826e20e90be4316f00"
server_key_version: "encryptionkey@0"
I20260812 09:46:12.819149 29783 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260812 09:46:12.821877 29807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:46:12.823271 29783 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 09:46:12.823395 29783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-1/wal
uuid: "3412b9c64bf3436db2fca81e7c9bdc85"
format_stamp: "Formatted at 2026-08-12 09:46:12 on dist-test-slave-jq4t"
server_key: "8614204776aed09b0119ebc3186946f7"
server_key_iv: "625ba1c8d61924826e20e90be4316f00"
server_key_version: "encryptionkey@0"
I20260812 09:46:12.823509 29783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:46:12.828109 29783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:46:12.828783 29783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:46:12.828969 29783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:46:12.829532 29783 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 09:46:12.830456 29783 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:46:12.830503 29783 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 09:46:12.830566 29783 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:46:12.830602 29783 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 09:46:12.839810 29783 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.2:39803
I20260812 09:46:12.839835 30005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.2:39803 every 8 connection(s)
I20260812 09:46:12.840781 29783 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-1/data/info.pb
I20260812 09:46:12.846521 29534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:46:12.842532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.2:46665 (local address 127.13.99.62:37945)
0812 09:46:12.842679 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0812 09:46:12.842683 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:46:12.843643 (+ 960us) server_negotiation.cc:407] Connection header received
0812 09:46:12.844980 (+ 1337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:46:12.844983 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:46:12.845029 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:46:12.845103 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:46:12.846131 (+ 1028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:46:12.846134 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:46:12.846136 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 09:46:12.846149 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:46:12.846170 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 09:46:12.846178 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:46:12.846254 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0812 09:46:12.846378 (+ 124us) server_negotiation.cc:299] Negotiation successful
0812 09:46:12.846417 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260812 09:46:12.847327 30008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:46:12.842857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:37945 (local address 127.13.99.2:46665)
0812 09:46:12.843499 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0812 09:46:12.843538 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0812 09:46:12.844788 (+ 1250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:46:12.845266 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:46:12.845272 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:46:12.845357 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:46:12.845864 (+ 507us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 09:46:12.845866 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:46:12.845903 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 09:46:12.845984 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:46:12.846256 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:46:12.846262 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:46:12.846267 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 09:46:12.846393 (+ 126us) client_negotiation.cc:240] Negotiation successful
0812 09:46:12.846572 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":133,"threads_started":1}
I20260812 09:46:12.848539 30006 heartbeater.cc:347] Connected to a master server at 127.13.99.62:37945
I20260812 09:46:12.848834 30006 heartbeater.cc:464] Registering TS with master...
I20260812 09:46:12.849092 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 29783
I20260812 09:46:12.849220 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-1/wal/instance
I20260812 09:46:12.849365 30006 heartbeater.cc:511] Master 127.13.99.62:37945 requested a full tablet report, sending...
I20260812 09:46:12.849550 13708 external_mini_cluster.cc:1428] Setting key ac3e0a6d5c84fab12b33c1e932436cdd
I20260812 09:46:12.850595 29459 ts_manager.cc:194] Registered new tserver with Master: 3412b9c64bf3436db2fca81e7c9bdc85 (127.13.99.2:39803)
I20260812 09:46:12.851567 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.99.3:0
--local_ip_for_outbound_sockets=127.13.99.3
--webserver_interface=127.13.99.3
--webserver_port=0
--tserver_master_addrs=127.13.99.62:37945
--builtin_ntp_servers=127.13.99.20:35733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260812 09:46:12.966717 30009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:46:12.966961 30009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:46:12.966998 30009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:46:12.971009 30009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:46:12.971058 30009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:46:12.971141 30009 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.99.3
I20260812 09:46:12.975780 30009 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:35733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.99.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.99.3
--webserver_port=0
--tserver_master_addrs=127.13.99.62:37945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.99.3
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:46:12.976819 30009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:46:12.978178 30009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:46:12.987757 30019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:46:12.987697 30016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:46:12.987717 30015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:46:12.988135 30009 server_base.cc:1034] running on GCE node
I20260812 09:46:12.988605 30009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:46:12.989238 30009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:46:12.990415 30009 hybrid_clock.cc:648] HybridClock initialized: now 1786527972990402 us; error 29 us; skew 500 ppm
I20260812 09:46:12.992401 30009 webserver.cc:460] Webserver started at http://127.13.99.3:34771/ using document root <none> and password file <none>
I20260812 09:46:12.993048 30009 fs_manager.cc:359] Metadata directory not provided
I20260812 09:46:12.993125 30009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:46:12.993351 30009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:46:12.995448 30009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-2/data/instance:
uuid: "4b938076afa6431a9fe4a020323a5ae6"
format_stamp: "Formatted at 2026-08-12 09:46:12 on dist-test-slave-jq4t"
server_key: "a6d7da96e57b221dacf8aa8db179ead2"
server_key_iv: "8f14718d308d4a3500ce3895bd68998b"
server_key_version: "encryptionkey@0"
I20260812 09:46:12.996047 30009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-2/wal/instance:
uuid: "4b938076afa6431a9fe4a020323a5ae6"
format_stamp: "Formatted at 2026-08-12 09:46:12 on dist-test-slave-jq4t"
server_key: "a6d7da96e57b221dacf8aa8db179ead2"
server_key_iv: "8f14718d308d4a3500ce3895bd68998b"
server_key_version: "encryptionkey@0"
I20260812 09:46:13.000056 30009 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260812 09:46:13.002753 30036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:46:13.004099 30009 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 09:46:13.004209 30009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-2/wal
uuid: "4b938076afa6431a9fe4a020323a5ae6"
format_stamp: "Formatted at 2026-08-12 09:46:12 on dist-test-slave-jq4t"
server_key: "a6d7da96e57b221dacf8aa8db179ead2"
server_key_iv: "8f14718d308d4a3500ce3895bd68998b"
server_key_version: "encryptionkey@0"
I20260812 09:46:13.004298 30009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:46:13.009006 30009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:46:13.009582 30009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:46:13.009719 30009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:46:13.010205 30009 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 09:46:13.011101 30009 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:46:13.011149 30009 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 09:46:13.011188 30009 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:46:13.011204 30009 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 09:46:13.020088 30009 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.3:42901
I20260812 09:46:13.020128 30232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.3:42901 every 8 connection(s)
I20260812 09:46:13.021112 30009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-2/data/info.pb
I20260812 09:46:13.027667 29534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:46:13.023416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.3:50637 (local address 127.13.99.62:37945)
0812 09:46:13.023577 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0812 09:46:13.023582 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:46:13.024767 (+ 1185us) server_negotiation.cc:407] Connection header received
0812 09:46:13.026066 (+ 1299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:46:13.026069 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:46:13.026108 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:46:13.026185 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:46:13.027216 (+ 1031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:46:13.027219 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:46:13.027221 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 09:46:13.027232 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:46:13.027253 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 09:46:13.027261 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:46:13.027356 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0812 09:46:13.027512 (+ 156us) server_negotiation.cc:299] Negotiation successful
0812 09:46:13.027563 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260812 09:46:13.028458 30235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:46:13.023888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:37945 (local address 127.13.99.3:50637)
0812 09:46:13.024624 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0812 09:46:13.024666 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 09:46:13.025870 (+ 1204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:46:13.026361 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:46:13.026368 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:46:13.026453 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:46:13.026963 (+ 510us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 09:46:13.026965 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:46:13.027004 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 09:46:13.027072 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:46:13.027364 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:46:13.027371 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:46:13.027376 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 09:46:13.027505 (+ 129us) client_negotiation.cc:240] Negotiation successful
0812 09:46:13.027689 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":119,"threads_started":1}
I20260812 09:46:13.028908 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 30009
I20260812 09:46:13.029050 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786527905902985-13708-0/minicluster-data/ts-2/wal/instance
I20260812 09:46:13.029402 13708 external_mini_cluster.cc:1428] Setting key 8cfdf0bccf51083786d280a79b53c0f8
I20260812 09:46:13.029858 30233 heartbeater.cc:347] Connected to a master server at 127.13.99.62:37945
I20260812 09:46:13.030157 30233 heartbeater.cc:464] Registering TS with master...
I20260812 09:46:13.030671 30233 heartbeater.cc:511] Master 127.13.99.62:37945 requested a full tablet report, sending...
I20260812 09:46:13.031759 29460 ts_manager.cc:194] Registered new tserver with Master: 4b938076afa6431a9fe4a020323a5ae6 (127.13.99.3:42901)
I20260812 09:46:13.043730 13708 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260812 09:46:13.046516 29534 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 09:46:13.045593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50008 (local address 127.13.99.62:37945)
0812 09:46:13.045741 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0812 09:46:13.045746 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:46:13.045957 (+ 211us) server_negotiation.cc:407] Connection header received
0812 09:46:13.046053 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:46:13.046055 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:46:13.046086 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:46:13.046141 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:46:13.046403 (+ 262us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50008: BlockingRecv error: recv got EOF from 127.0.0.1:50008 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260812 09:46:13.046667 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 29562
I20260812 09:46:13.060279 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 29783
I20260812 09:46:13.084673 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 30009
I20260812 09:46:13.094588 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 29412
2026-08-12T09:46:13Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (862 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 09:46:13 dist-test-slave-jq4t krb5kdc[30249](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 09:46:13 dist-test-slave-jq4t krb5kdc[30249](info): set up 2 sockets
Aug 12 09:46:13 dist-test-slave-jq4t krb5kdc[30249](info): commencing operation
krb5kdc: starting...
W20260812 09:46:14.386682 13708 mini_kdc.cc:121] Time spent starting KDC: real 1.263s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/krb5kdc/test-user.keytab.
Aug 12 09:46:14 dist-test-slave-jq4t krb5kdc[30249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527974, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T09:46:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T09:46:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.99.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.99.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.99.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab.
Entry for principal kudu/127.13.99.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab.
I20260812 09:46:14.516801 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.62:41023
--webserver_interface=127.13.99.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab
--principal=kudu/127.13.99.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:37935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.99.62:41023
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:46:14.632915 30270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:46:14.633211 30270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:46:14.633287 30270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:46:14.637373 30270 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 09:46:14.637493 30270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:46:14.637533 30270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:46:14.637578 30270 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 09:46:14.637610 30270 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 09:46:14.642875 30270 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:37935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.99.62:41023
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab
--principal=kudu/127.13.99.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.62:41023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.99.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30270
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:46:14.643953 30270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:46:14.645051 30270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:46:14.653851 30283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:46:14.653857 30287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:46:14.653900 30284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:46:14.654160 30270 server_base.cc:1034] running on GCE node
I20260812 09:46:14.654757 30270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:46:14.655817 30270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:46:14.656976 30270 hybrid_clock.cc:648] HybridClock initialized: now 1786527974656944 us; error 45 us; skew 500 ppm
Aug 12 09:46:14 dist-test-slave-jq4t krb5kdc[30249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527974, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:46:14.659920 30270 init.cc:377] Logged in from keytab as kudu/127.13.99.62@KRBTEST.COM (short username kudu)
I20260812 09:46:14.660908 30270 webserver.cc:460] Webserver started at http://127.13.99.62:35003/ using document root <none> and password file <none>
I20260812 09:46:14.661495 30270 fs_manager.cc:359] Metadata directory not provided
I20260812 09:46:14.661593 30270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:46:14.661835 30270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:46:14.663892 30270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/minicluster-data/master-0/data/instance:
uuid: "60b9bd9e33d44a91bbdb4c13ef55bf42"
format_stamp: "Formatted at 2026-08-12 09:46:14 on dist-test-slave-jq4t"
server_key: "d0fe49f6dfa05ced45081dc67c2a6257"
server_key_iv: "b82bbc99dfac964b466f8800fd36c5d0"
server_key_version: "encryptionkey@0"
I20260812 09:46:14.664463 30270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/minicluster-data/master-0/wal/instance:
uuid: "60b9bd9e33d44a91bbdb4c13ef55bf42"
format_stamp: "Formatted at 2026-08-12 09:46:14 on dist-test-slave-jq4t"
server_key: "d0fe49f6dfa05ced45081dc67c2a6257"
server_key_iv: "b82bbc99dfac964b466f8800fd36c5d0"
server_key_version: "encryptionkey@0"
I20260812 09:46:14.668184 30270 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260812 09:46:14.670689 30301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:46:14.671860 30270 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260812 09:46:14.672006 30270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/minicluster-data/master-0/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/minicluster-data/master-0/wal
uuid: "60b9bd9e33d44a91bbdb4c13ef55bf42"
format_stamp: "Formatted at 2026-08-12 09:46:14 on dist-test-slave-jq4t"
server_key: "d0fe49f6dfa05ced45081dc67c2a6257"
server_key_iv: "b82bbc99dfac964b466f8800fd36c5d0"
server_key_version: "encryptionkey@0"
I20260812 09:46:14.672118 30270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:46:14.692889 30270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:46:14.695719 30270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:46:14.695923 30270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:46:14.703220 30270 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.62:41023
I20260812 09:46:14.703524 30403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.62:41023 every 8 connection(s)
I20260812 09:46:14.704213 30270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
I20260812 09:46:14.707077 30404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:46:14.712587 30404 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60b9bd9e33d44a91bbdb4c13ef55bf42: Bootstrap starting.
I20260812 09:46:14.714344 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 30270
I20260812 09:46:14.714494 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/minicluster-data/master-0/wal/instance
I20260812 09:46:14.714771 13708 external_mini_cluster.cc:1428] Setting key fad463dcf58a76c76f2237ec5600487d
I20260812 09:46:14.715070 30404 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 60b9bd9e33d44a91bbdb4c13ef55bf42: Neither blocks nor log segments found. Creating new log.
I20260812 09:46:14.715960 30404 log.cc:824] T 00000000000000000000000000000000 P 60b9bd9e33d44a91bbdb4c13ef55bf42: Log is configured to *not* fsync() on all Append() calls
I20260812 09:46:14.718122 30404 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60b9bd9e33d44a91bbdb4c13ef55bf42: No bootstrap required, opened a new log
I20260812 09:46:14.722211 30404 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60b9bd9e33d44a91bbdb4c13ef55bf42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60b9bd9e33d44a91bbdb4c13ef55bf42" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 41023 } }
I20260812 09:46:14.722506 30404 raft_consensus.cc:374] T 00000000000000000000000000000000 P 60b9bd9e33d44a91bbdb4c13ef55bf42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:46:14.722592 30404 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60b9bd9e33d44a91bbdb4c13ef55bf42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b9bd9e33d44a91bbdb4c13ef55bf42, State: Initialized, Role: FOLLOWER
I20260812 09:46:14.723151 30404 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60b9bd9e33d44a91bbdb4c13ef55bf42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60b9bd9e33d44a91bbdb4c13ef55bf42" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 41023 } }
I20260812 09:46:14.723332 30404 raft_consensus.cc:388] T 00000000000000000000000000000000 P 60b9bd9e33d44a91bbdb4c13ef55bf42 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 09:46:14.723407 30404 raft_consensus.cc:482] T 00000000000000000000000000000000 P 60b9bd9e33d44a91bbdb4c13ef55bf42 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 09:46:14.723537 30404 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60b9bd9e33d44a91bbdb4c13ef55bf42 [term 0 FOLLOWER]: Advancing to term 1
Aug 12 09:46:14 dist-test-slave-jq4t krb5kdc[30249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527974, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
I20260812 09:46:14.724743 30404 raft_consensus.cc:504] T 00000000000000000000000000000000 P 60b9bd9e33d44a91bbdb4c13ef55bf42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60b9bd9e33d44a91bbdb4c13ef55bf42" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 41023 } }
I20260812 09:46:14.725131 30404 leader_election.cc:302] T 00000000000000000000000000000000 P 60b9bd9e33d44a91bbdb4c13ef55bf42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 60b9bd9e33d44a91bbdb4c13ef55bf42; no voters:
I20260812 09:46:14.725497 30404 leader_election.cc:288] T 00000000000000000000000000000000 P 60b9bd9e33d44a91bbdb4c13ef55bf42 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 09:46:14.725576 30412 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 60b9bd9e33d44a91bbdb4c13ef55bf42 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 09:46:14.725785 30412 raft_consensus.cc:686] T 00000000000000000000000000000000 P 60b9bd9e33d44a91bbdb4c13ef55bf42 [term 1 LEADER]: Becoming Leader. State: Replica: 60b9bd9e33d44a91bbdb4c13ef55bf42, State: Running, Role: LEADER
I20260812 09:46:14.726082 30412 consensus_queue.cc:237] T 00000000000000000000000000000000 P 60b9bd9e33d44a91bbdb4c13ef55bf42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60b9bd9e33d44a91bbdb4c13ef55bf42" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 41023 } }
I20260812 09:46:14.726866 30404 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60b9bd9e33d44a91bbdb4c13ef55bf42 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 09:46:14.727761 30414 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60b9bd9e33d44a91bbdb4c13ef55bf42 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 60b9bd9e33d44a91bbdb4c13ef55bf42. Latest consensus state: current_term: 1 leader_uuid: "60b9bd9e33d44a91bbdb4c13ef55bf42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60b9bd9e33d44a91bbdb4c13ef55bf42" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 41023 } } }
I20260812 09:46:14.727921 30414 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60b9bd9e33d44a91bbdb4c13ef55bf42 [sys.catalog]: This master's current role is: LEADER
I20260812 09:46:14.728368 30429 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 09:46:14.729831 30413 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60b9bd9e33d44a91bbdb4c13ef55bf42 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "60b9bd9e33d44a91bbdb4c13ef55bf42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60b9bd9e33d44a91bbdb4c13ef55bf42" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 41023 } } }
I20260812 09:46:14.729924 30413 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60b9bd9e33d44a91bbdb4c13ef55bf42 [sys.catalog]: This master's current role is: LEADER
I20260812 09:46:14.731141 30408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:46:14.715909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41356 (local address 127.13.99.62:41023)
0812 09:46:14.716351 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0812 09:46:14.716391 (+ 40us) server_negotiation.cc:399] Waiting for connection header
0812 09:46:14.716439 (+ 48us) server_negotiation.cc:407] Connection header received
0812 09:46:14.717230 (+ 791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:46:14.717250 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:46:14.717579 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:46:14.717859 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:46:14.718827 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:46:14.719859 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:46:14.721262 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:46:14.722032 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:46:14.722182 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:46:14.725407 (+ 3225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:46:14.725412 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:46:14.725426 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:46:14.725453 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:46:14.727781 (+ 2328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:46:14.728352 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:46:14.728358 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:46:14.728364 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:46:14.728556 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:46:14.728899 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:46:14.728902 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:46:14.728904 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:46:14.729456 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:46:14.729679 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0812 09:46:14.730018 (+ 339us) server_negotiation.cc:299] Negotiation successful
0812 09:46:14.730366 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":136,"threads_started":1}
I20260812 09:46:14.731833 30429 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 09:46:14.738368 30429 catalog_manager.cc:1347] Generated new cluster ID: 7fa079d829e34e0881a1db62a1ee3fbb
I20260812 09:46:14.738435 30429 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 09:46:14.757895 30429 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 09:46:14.758958 30429 catalog_manager.cc:1504] Loading token signing keys...
I20260812 09:46:14.773784 30429 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 60b9bd9e33d44a91bbdb4c13ef55bf42: Generated new TSK 0
I20260812 09:46:14.774760 30429 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.99.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.99.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.99.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.1.keytab.
Entry for principal kudu/127.13.99.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.1.keytab.
I20260812 09:46:14.819538 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.99.1:0
--local_ip_for_outbound_sockets=127.13.99.1
--webserver_interface=127.13.99.1
--webserver_port=0
--tserver_master_addrs=127.13.99.62:41023
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.1.keytab
--principal=kudu/127.13.99.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:37935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:46:14.962843 30446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:46:14.963140 30446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:46:14.963217 30446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:46:14.967394 30446 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:46:14.967481 30446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:46:14.967578 30446 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.99.1
I20260812 09:46:14.972930 30446 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:37935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.1.keytab
--principal=kudu/127.13.99.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.99.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.99.1
--webserver_port=0
--tserver_master_addrs=127.13.99.62:41023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30446
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.99.1
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:46:14.974056 30446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:46:14.975229 30446 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:46:14.985117 30457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:46:14.985158 30458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:46:14.985410 30460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:46:14.985564 30446 server_base.cc:1034] running on GCE node
I20260812 09:46:14.986109 30446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:46:14.986749 30446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:46:14.987944 30446 hybrid_clock.cc:648] HybridClock initialized: now 1786527974987936 us; error 36 us; skew 500 ppm
Aug 12 09:46:14 dist-test-slave-jq4t krb5kdc[30249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527974, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:46:14.993592 30446 init.cc:377] Logged in from keytab as kudu/127.13.99.1@KRBTEST.COM (short username kudu)
I20260812 09:46:14.994500 30446 webserver.cc:460] Webserver started at http://127.13.99.1:46647/ using document root <none> and password file <none>
I20260812 09:46:14.995098 30446 fs_manager.cc:359] Metadata directory not provided
I20260812 09:46:14.995196 30446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:46:14.995450 30446 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:46:14.997613 30446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/minicluster-data/ts-0/data/instance:
uuid: "fda0e5942037408185325d3fe00eea11"
format_stamp: "Formatted at 2026-08-12 09:46:14 on dist-test-slave-jq4t"
server_key: "791d9e7f1f7795a13e47db5779371edc"
server_key_iv: "81e967deeed4542f3e847187061b32a6"
server_key_version: "encryptionkey@0"
I20260812 09:46:14.998198 30446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/minicluster-data/ts-0/wal/instance:
uuid: "fda0e5942037408185325d3fe00eea11"
format_stamp: "Formatted at 2026-08-12 09:46:14 on dist-test-slave-jq4t"
server_key: "791d9e7f1f7795a13e47db5779371edc"
server_key_iv: "81e967deeed4542f3e847187061b32a6"
server_key_version: "encryptionkey@0"
I20260812 09:46:15.001982 30446 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 09:46:15.004426 30475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:46:15.005707 30446 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260812 09:46:15.005841 30446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/minicluster-data/ts-0/wal
uuid: "fda0e5942037408185325d3fe00eea11"
format_stamp: "Formatted at 2026-08-12 09:46:14 on dist-test-slave-jq4t"
server_key: "791d9e7f1f7795a13e47db5779371edc"
server_key_iv: "81e967deeed4542f3e847187061b32a6"
server_key_version: "encryptionkey@0"
I20260812 09:46:15.005950 30446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:46:15.037906 30446 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:46:15.040916 30446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:46:15.041126 30446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:46:15.041742 30446 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 09:46:15.042599 30446 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:46:15.042675 30446 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 09:46:15.042747 30446 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:46:15.042789 30446 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 09:46:15.051734 30446 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.1:42519
I20260812 09:46:15.051761 30671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.1:42519 every 8 connection(s)
I20260812 09:46:15.052640 30446 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/minicluster-data/ts-0/data/info.pb
I20260812 09:46:15.058086 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 30446
I20260812 09:46:15.058224 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/minicluster-data/ts-0/wal/instance
I20260812 09:46:15.058530 13708 external_mini_cluster.cc:1428] Setting key 5337b455355dbf8b146df17d531d34f6
Aug 12 09:46:15 dist-test-slave-jq4t krb5kdc[30249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527974, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.1@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
I20260812 09:46:15.067152 30408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:46:15.054557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.1:55203 (local address 127.13.99.62:41023)
0812 09:46:15.054702 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0812 09:46:15.054706 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:46:15.055505 (+ 799us) server_negotiation.cc:407] Connection header received
0812 09:46:15.056997 (+ 1492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:46:15.057000 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:46:15.057060 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:46:15.057164 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:46:15.058447 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:46:15.059180 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:46:15.060583 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:46:15.060840 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:46:15.060950 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:46:15.063443 (+ 2493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:46:15.063446 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:46:15.063449 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:46:15.063481 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:46:15.065075 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:46:15.065594 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:46:15.065598 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:46:15.065600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:46:15.065690 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:46:15.066060 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:46:15.066063 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:46:15.066065 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:46:15.066255 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:46:15.066372 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0812 09:46:15.066851 (+ 479us) server_negotiation.cc:299] Negotiation successful
0812 09:46:15.066980 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260812 09:46:15.067919 30675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:46:15.054917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:41023 (local address 127.13.99.1:55203)
0812 09:46:15.055352 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0812 09:46:15.055402 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0812 09:46:15.056773 (+ 1371us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:46:15.057344 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:46:15.057350 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:46:15.057692 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:46:15.058275 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:46:15.058308 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:46:15.059310 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:46:15.059315 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:46:15.060433 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:46:15.060444 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:46:15.061001 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:46:15.061006 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:46:15.061176 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:46:15.061805 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:46:15.061821 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:46:15.063273 (+ 1452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:46:15.065191 (+ 1918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:46:15.065197 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:46:15.065211 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:46:15.065484 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:46:15.065796 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:46:15.065799 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:46:15.065801 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:46:15.065951 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:46:15.066366 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:46:15.066372 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:46:15.066616 (+ 244us) client_negotiation.cc:769] Sending connection context
0812 09:46:15.066809 (+ 193us) client_negotiation.cc:240] Negotiation successful
0812 09:46:15.067125 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":113,"threads_started":1}
WARNING: no policy specified for kudu/127.13.99.2@KRBTEST.COM; defaulting to no policy
I20260812 09:46:15.069278 30672 heartbeater.cc:347] Connected to a master server at 127.13.99.62:41023
I20260812 09:46:15.069518 30672 heartbeater.cc:464] Registering TS with master...
I20260812 09:46:15.070070 30672 heartbeater.cc:511] Master 127.13.99.62:41023 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.99.2@KRBTEST.COM" created.
I20260812 09:46:15.071614 30329 ts_manager.cc:194] Registered new tserver with Master: fda0e5942037408185325d3fe00eea11 (127.13.99.1:42519)
I20260812 09:46:15.072835 30329 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.99.1@KRBTEST.COM'} at 127.13.99.1:55203
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.99.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.2.keytab.
Entry for principal kudu/127.13.99.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.2.keytab.
I20260812 09:46:15.086984 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.99.2:0
--local_ip_for_outbound_sockets=127.13.99.2
--webserver_interface=127.13.99.2
--webserver_port=0
--tserver_master_addrs=127.13.99.62:41023
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.2.keytab
--principal=kudu/127.13.99.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:37935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:46:15.197928 30681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:46:15.198186 30681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:46:15.198225 30681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:46:15.202114 30681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:46:15.202164 30681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:46:15.202219 30681 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.99.2
I20260812 09:46:15.207244 30681 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:37935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.2.keytab
--principal=kudu/127.13.99.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.99.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.99.2
--webserver_port=0
--tserver_master_addrs=127.13.99.62:41023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30681
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.99.2
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:46:15.208259 30681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:46:15.209334 30681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:46:15.219097 30692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:46:15.219103 30688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:46:15.219106 30689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:46:15.219308 30681 server_base.cc:1034] running on GCE node
I20260812 09:46:15.219782 30681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:46:15.220422 30681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:46:15.221647 30681 hybrid_clock.cc:648] HybridClock initialized: now 1786527975221618 us; error 47 us; skew 500 ppm
Aug 12 09:46:15 dist-test-slave-jq4t krb5kdc[30249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527975, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:46:15.224431 30681 init.cc:377] Logged in from keytab as kudu/127.13.99.2@KRBTEST.COM (short username kudu)
I20260812 09:46:15.225330 30681 webserver.cc:460] Webserver started at http://127.13.99.2:34381/ using document root <none> and password file <none>
I20260812 09:46:15.225883 30681 fs_manager.cc:359] Metadata directory not provided
I20260812 09:46:15.225986 30681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:46:15.226219 30681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:46:15.228457 30681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/minicluster-data/ts-1/data/instance:
uuid: "0e60857e6df448cdbf90e91888fcded2"
format_stamp: "Formatted at 2026-08-12 09:46:15 on dist-test-slave-jq4t"
server_key: "072cbf3d2befa38a3c68b6c2b6000c14"
server_key_iv: "a14c00f0b4d4e3d0ce95a621a80f5204"
server_key_version: "encryptionkey@0"
I20260812 09:46:15.229081 30681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/minicluster-data/ts-1/wal/instance:
uuid: "0e60857e6df448cdbf90e91888fcded2"
format_stamp: "Formatted at 2026-08-12 09:46:15 on dist-test-slave-jq4t"
server_key: "072cbf3d2befa38a3c68b6c2b6000c14"
server_key_iv: "a14c00f0b4d4e3d0ce95a621a80f5204"
server_key_version: "encryptionkey@0"
I20260812 09:46:15.232801 30681 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 09:46:15.235152 30703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:46:15.236670 30681 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 09:46:15.236800 30681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/minicluster-data/ts-1/wal
uuid: "0e60857e6df448cdbf90e91888fcded2"
format_stamp: "Formatted at 2026-08-12 09:46:15 on dist-test-slave-jq4t"
server_key: "072cbf3d2befa38a3c68b6c2b6000c14"
server_key_iv: "a14c00f0b4d4e3d0ce95a621a80f5204"
server_key_version: "encryptionkey@0"
I20260812 09:46:15.236927 30681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:46:15.248123 30681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:46:15.250897 30681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:46:15.251091 30681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:46:15.251657 30681 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 09:46:15.252470 30681 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:46:15.252600 30681 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 09:46:15.252665 30681 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:46:15.252712 30681 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 09:46:15.261996 30681 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.2:43747
I20260812 09:46:15.262042 30886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.2:43747 every 8 connection(s)
I20260812 09:46:15.262904 30681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/minicluster-data/ts-1/data/info.pb
I20260812 09:46:15.263332 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 30681
I20260812 09:46:15.263478 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/minicluster-data/ts-1/wal/instance
I20260812 09:46:15.263808 13708 external_mini_cluster.cc:1428] Setting key 2d06951701c589a016429ce89c2a263e
Aug 12 09:46:15 dist-test-slave-jq4t krb5kdc[30249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527975, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.2@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.99.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.99.3@KRBTEST.COM" created.
I20260812 09:46:15.279392 30408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:46:15.264895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.2:50937 (local address 127.13.99.62:41023)
0812 09:46:15.265041 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0812 09:46:15.265046 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:46:15.266070 (+ 1024us) server_negotiation.cc:407] Connection header received
0812 09:46:15.267529 (+ 1459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:46:15.267531 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:46:15.267576 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:46:15.267645 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:46:15.269302 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:46:15.269836 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:46:15.271746 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:46:15.271970 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:46:15.272061 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:46:15.275062 (+ 3001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:46:15.275066 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:46:15.275069 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:46:15.275101 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:46:15.276909 (+ 1808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:46:15.277469 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:46:15.277476 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:46:15.277478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:46:15.277572 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:46:15.277934 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:46:15.277955 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:46:15.277956 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:46:15.278217 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:46:15.278299 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0812 09:46:15.278975 (+ 676us) server_negotiation.cc:299] Negotiation successful
0812 09:46:15.279124 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260812 09:46:15.280011 30889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:46:15.265397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:41023 (local address 127.13.99.2:50937)
0812 09:46:15.265906 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0812 09:46:15.265947 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0812 09:46:15.267289 (+ 1342us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:46:15.267832 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:46:15.267844 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:46:15.268386 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:46:15.269135 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:46:15.269159 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:46:15.269979 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:46:15.269994 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:46:15.271608 (+ 1614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:46:15.271622 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:46:15.272147 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:46:15.272154 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:46:15.272401 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:46:15.273114 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:46:15.273138 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:46:15.274892 (+ 1754us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:46:15.277053 (+ 2161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:46:15.277061 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:46:15.277076 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:46:15.277337 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:46:15.277698 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:46:15.277701 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:46:15.277702 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:46:15.277844 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:46:15.278321 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:46:15.278331 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:46:15.278672 (+ 341us) client_negotiation.cc:769] Sending connection context
0812 09:46:15.278865 (+ 193us) client_negotiation.cc:240] Negotiation successful
0812 09:46:15.279141 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":157,"threads_started":1}
I20260812 09:46:15.281419 30887 heartbeater.cc:347] Connected to a master server at 127.13.99.62:41023
I20260812 09:46:15.281693 30887 heartbeater.cc:464] Registering TS with master...
I20260812 09:46:15.282330 30887 heartbeater.cc:511] Master 127.13.99.62:41023 requested a full tablet report, sending...
I20260812 09:46:15.283562 30329 ts_manager.cc:194] Registered new tserver with Master: 0e60857e6df448cdbf90e91888fcded2 (127.13.99.2:43747)
I20260812 09:46:15.284209 30329 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.99.2@KRBTEST.COM'} at 127.13.99.2:50937
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.99.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.3.keytab.
Entry for principal kudu/127.13.99.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.3.keytab.
I20260812 09:46:15.295801 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.99.3:0
--local_ip_for_outbound_sockets=127.13.99.3
--webserver_interface=127.13.99.3
--webserver_port=0
--tserver_master_addrs=127.13.99.62:41023
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.3.keytab
--principal=kudu/127.13.99.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:37935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:46:15.411085 30898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:46:15.411370 30898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:46:15.411437 30898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:46:15.415652 30898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:46:15.415730 30898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:46:15.415817 30898 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.99.3
I20260812 09:46:15.421085 30898 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:37935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.3.keytab
--principal=kudu/127.13.99.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.99.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.99.3
--webserver_port=0
--tserver_master_addrs=127.13.99.62:41023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30898
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.99.3
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:46:15.422128 30898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:46:15.423161 30898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:46:15.432905 30907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:46:15.432924 30912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:46:15.432919 30909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:46:15.433240 30898 server_base.cc:1034] running on GCE node
I20260812 09:46:15.433663 30898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:46:15.434391 30898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:46:15.435586 30898 hybrid_clock.cc:648] HybridClock initialized: now 1786527975435569 us; error 26 us; skew 500 ppm
Aug 12 09:46:15 dist-test-slave-jq4t krb5kdc[30249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527975, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:46:15.438390 30898 init.cc:377] Logged in from keytab as kudu/127.13.99.3@KRBTEST.COM (short username kudu)
I20260812 09:46:15.439268 30898 webserver.cc:460] Webserver started at http://127.13.99.3:33781/ using document root <none> and password file <none>
I20260812 09:46:15.439843 30898 fs_manager.cc:359] Metadata directory not provided
I20260812 09:46:15.439939 30898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:46:15.440196 30898 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:46:15.442381 30898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/minicluster-data/ts-2/data/instance:
uuid: "f7027607aa404406bd3c9b9e935bc9f7"
format_stamp: "Formatted at 2026-08-12 09:46:15 on dist-test-slave-jq4t"
server_key: "be26d6e2573d54529c331e789fe37725"
server_key_iv: "7461ed6c9348654478b2f39609844cb6"
server_key_version: "encryptionkey@0"
I20260812 09:46:15.442971 30898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/minicluster-data/ts-2/wal/instance:
uuid: "f7027607aa404406bd3c9b9e935bc9f7"
format_stamp: "Formatted at 2026-08-12 09:46:15 on dist-test-slave-jq4t"
server_key: "be26d6e2573d54529c331e789fe37725"
server_key_iv: "7461ed6c9348654478b2f39609844cb6"
server_key_version: "encryptionkey@0"
I20260812 09:46:15.446758 30898 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 09:46:15.449338 30922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:46:15.450590 30898 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 09:46:15.450742 30898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/minicluster-data/ts-2/wal
uuid: "f7027607aa404406bd3c9b9e935bc9f7"
format_stamp: "Formatted at 2026-08-12 09:46:15 on dist-test-slave-jq4t"
server_key: "be26d6e2573d54529c331e789fe37725"
server_key_iv: "7461ed6c9348654478b2f39609844cb6"
server_key_version: "encryptionkey@0"
I20260812 09:46:15.450872 30898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:46:15.474377 30898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:46:15.477509 30898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:46:15.477717 30898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:46:15.478327 30898 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 09:46:15.479174 30898 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:46:15.479249 30898 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 09:46:15.479321 30898 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:46:15.479367 30898 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 09:46:15.488504 30898 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.3:37439
I20260812 09:46:15.488518 31111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.3:37439 every 8 connection(s)
I20260812 09:46:15.489400 30898 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/minicluster-data/ts-2/data/info.pb
I20260812 09:46:15.493083 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 30898
I20260812 09:46:15.493224 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/minicluster-data/ts-2/wal/instance
I20260812 09:46:15.493551 13708 external_mini_cluster.cc:1428] Setting key 940cfcc87d177e78b6193452b5c95d0f
Aug 12 09:46:15 dist-test-slave-jq4t krb5kdc[30249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527975, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.3@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
I20260812 09:46:15.503305 30408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:46:15.491237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.3:56993 (local address 127.13.99.62:41023)
0812 09:46:15.491398 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0812 09:46:15.491413 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 09:46:15.492161 (+ 748us) server_negotiation.cc:407] Connection header received
0812 09:46:15.493963 (+ 1802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:46:15.493967 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:46:15.494042 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:46:15.494134 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:46:15.495613 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:46:15.496197 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:46:15.497439 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:46:15.497657 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:46:15.497735 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:46:15.500105 (+ 2370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:46:15.500125 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:46:15.500127 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:46:15.500152 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:46:15.501596 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:46:15.502049 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:46:15.502052 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:46:15.502053 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:46:15.502127 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:46:15.502411 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:46:15.502413 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:46:15.502414 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:46:15.502582 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:46:15.502656 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0812 09:46:15.503095 (+ 439us) server_negotiation.cc:299] Negotiation successful
0812 09:46:15.503186 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260812 09:46:15.504079 31115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:46:15.491559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:41023 (local address 127.13.99.3:56993)
0812 09:46:15.492008 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0812 09:46:15.492049 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0812 09:46:15.493715 (+ 1666us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:46:15.494337 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:46:15.494344 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:46:15.494895 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:46:15.495457 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:46:15.495478 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:46:15.496328 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:46:15.496332 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:46:15.497320 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:46:15.497330 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:46:15.497784 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:46:15.497786 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:46:15.497928 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:46:15.498493 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:46:15.498509 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:46:15.499941 (+ 1432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:46:15.501700 (+ 1759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:46:15.501705 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:46:15.501716 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:46:15.501954 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:46:15.502205 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:46:15.502208 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:46:15.502209 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:46:15.502334 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:46:15.502668 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:46:15.502673 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:46:15.502886 (+ 213us) client_negotiation.cc:769] Sending connection context
0812 09:46:15.503038 (+ 152us) client_negotiation.cc:240] Negotiation successful
0812 09:46:15.503273 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":121,"threads_started":1}
I20260812 09:46:15.505208 31113 heartbeater.cc:347] Connected to a master server at 127.13.99.62:41023
I20260812 09:46:15.505409 31113 heartbeater.cc:464] Registering TS with master...
I20260812 09:46:15.505860 31113 heartbeater.cc:511] Master 127.13.99.62:41023 requested a full tablet report, sending...
I20260812 09:46:15.506939 30329 ts_manager.cc:194] Registered new tserver with Master: f7027607aa404406bd3c9b9e935bc9f7 (127.13.99.3:37439)
I20260812 09:46:15.507750 30329 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.99.3@KRBTEST.COM'} at 127.13.99.3:56993
I20260812 09:46:15.508421 13708 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 09:46:15.517408 30408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:46:15.510485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39948 (local address 127.13.99.62:41023)
0812 09:46:15.510630 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0812 09:46:15.510634 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:46:15.510811 (+ 177us) server_negotiation.cc:407] Connection header received
0812 09:46:15.510970 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:46:15.510973 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:46:15.511012 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:46:15.511084 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:46:15.511780 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:46:15.512255 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:46:15.513108 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:46:15.513310 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:46:15.513385 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:46:15.514576 (+ 1191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:46:15.514579 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:46:15.514580 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:46:15.514600 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:46:15.515967 (+ 1367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:46:15.516337 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:46:15.516339 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:46:15.516341 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:46:15.516410 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:46:15.516718 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:46:15.516720 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:46:15.516721 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:46:15.516912 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:46:15.517017 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0812 09:46:15.517215 (+ 198us) server_negotiation.cc:299] Negotiation successful
0812 09:46:15.517305 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260812 09:46:15.520141 30329 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39948:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 09:46:15.523285 30329 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 09:46:15.538676 31136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:46:15.532285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.3:37439 (local address 127.0.0.1:45984)
0812 09:46:15.533035 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0812 09:46:15.533050 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 09:46:15.533151 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:46:15.534262 (+ 1111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:46:15.534265 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:46:15.534280 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:46:15.534373 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:46:15.534387 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:46:15.535448 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:46:15.535454 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:46:15.537009 (+ 1555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:46:15.537021 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:46:15.538103 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:46:15.538110 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:46:15.538217 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:46:15.538252 (+ 35us) client_negotiation.cc:769] Sending connection context
0812 09:46:15.538379 (+ 127us) client_negotiation.cc:240] Negotiation successful
0812 09:46:15.538506 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":682,"mutex_wait_us":113,"spinlock_wait_cycles":1664,"thread_start_us":140,"threads_started":1}
I20260812 09:46:15.539026 31135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:46:15.532769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.1:42519 (local address 127.0.0.1:47108)
0812 09:46:15.533035 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0812 09:46:15.533050 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 09:46:15.533150 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:46:15.533943 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:46:15.533950 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:46:15.533973 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:46:15.534117 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:46:15.534124 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:46:15.535648 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:46:15.535652 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:46:15.537018 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:46:15.537025 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:46:15.538417 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:46:15.538421 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:46:15.538527 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:46:15.538550 (+ 23us) client_negotiation.cc:769] Sending connection context
0812 09:46:15.538635 (+ 85us) client_negotiation.cc:240] Negotiation successful
0812 09:46:15.538712 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"mutex_wait_us":123,"spinlock_wait_cycles":7168,"thread_start_us":118,"threads_started":1}
I20260812 09:46:15.539227 31133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:46:15.532046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.2:43747 (local address 127.0.0.1:43276)
0812 09:46:15.532600 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0812 09:46:15.532651 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0812 09:46:15.532776 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:46:15.534116 (+ 1340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:46:15.534119 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:46:15.534138 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:46:15.534262 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:46:15.534269 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:46:15.535648 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:46:15.535652 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:46:15.537325 (+ 1673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:46:15.537337 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:46:15.538353 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:46:15.538356 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:46:15.538482 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:46:15.538504 (+ 22us) client_negotiation.cc:769] Sending connection context
0812 09:46:15.538571 (+ 67us) client_negotiation.cc:240] Negotiation successful
0812 09:46:15.538629 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"mutex_wait_us":32,"thread_start_us":70,"threads_started":1}
I20260812 09:46:15.539418 31141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:46:15.533418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45984 (local address 127.13.99.3:37439)
0812 09:46:15.533794 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0812 09:46:15.533804 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 09:46:15.533826 (+ 22us) server_negotiation.cc:407] Connection header received
0812 09:46:15.533932 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:46:15.533937 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:46:15.534112 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:46:15.534497 (+ 385us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:46:15.534613 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:46:15.535312 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:46:15.537141 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:46:15.537966 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:46:15.538084 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:46:15.538231 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0812 09:46:15.539177 (+ 946us) server_negotiation.cc:299] Negotiation successful
0812 09:46:15.539276 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":160,"threads_started":1}
I20260812 09:46:15.539616 31138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:46:15.533049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47108 (local address 127.13.99.1:42519)
0812 09:46:15.533429 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0812 09:46:15.533443 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 09:46:15.533470 (+ 27us) server_negotiation.cc:407] Connection header received
0812 09:46:15.533556 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:46:15.533564 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:46:15.533781 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:46:15.533948 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:46:15.534239 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:46:15.535482 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:46:15.537501 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:46:15.538286 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:46:15.538424 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:46:15.538557 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0812 09:46:15.538672 (+ 115us) server_negotiation.cc:299] Negotiation successful
0812 09:46:15.538763 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":102,"threads_started":1}
I20260812 09:46:15.539444 31134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:46:15.532239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43276 (local address 127.13.99.2:43747)
0812 09:46:15.533428 (+ 1189us) server_negotiation.cc:206] Beginning negotiation
0812 09:46:15.533445 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 09:46:15.533470 (+ 25us) server_negotiation.cc:407] Connection header received
0812 09:46:15.533555 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:46:15.533560 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:46:15.533775 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:46:15.533940 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:46:15.534395 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:46:15.535508 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:46:15.537453 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:46:15.538011 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:46:15.538143 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:46:15.538287 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0812 09:46:15.538859 (+ 572us) server_negotiation.cc:299] Negotiation successful
0812 09:46:15.538981 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1058,"thread_start_us":187,"threads_started":1}
I20260812 09:46:15.542610 30783 tablet_service.cc:1467] Processing CreateTablet for tablet 64f571599c984eb59ebd74a7c2327a21 (DEFAULT_TABLE table=test-table [id=5e42e2f835314c6493b7f9a041ceda86]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 09:46:15.542608 31008 tablet_service.cc:1467] Processing CreateTablet for tablet 64f571599c984eb59ebd74a7c2327a21 (DEFAULT_TABLE table=test-table [id=5e42e2f835314c6493b7f9a041ceda86]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 09:46:15.542609 30551 tablet_service.cc:1467] Processing CreateTablet for tablet 64f571599c984eb59ebd74a7c2327a21 (DEFAULT_TABLE table=test-table [id=5e42e2f835314c6493b7f9a041ceda86]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 09:46:15.543689 30551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64f571599c984eb59ebd74a7c2327a21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:46:15.543694 31008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64f571599c984eb59ebd74a7c2327a21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:46:15.543694 30783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64f571599c984eb59ebd74a7c2327a21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:46:15.549820 31142 tablet_bootstrap.cc:492] T 64f571599c984eb59ebd74a7c2327a21 P 0e60857e6df448cdbf90e91888fcded2: Bootstrap starting.
I20260812 09:46:15.551467 31143 tablet_bootstrap.cc:492] T 64f571599c984eb59ebd74a7c2327a21 P fda0e5942037408185325d3fe00eea11: Bootstrap starting.
I20260812 09:46:15.552708 31145 tablet_bootstrap.cc:492] T 64f571599c984eb59ebd74a7c2327a21 P f7027607aa404406bd3c9b9e935bc9f7: Bootstrap starting.
I20260812 09:46:15.553285 31142 tablet_bootstrap.cc:654] T 64f571599c984eb59ebd74a7c2327a21 P 0e60857e6df448cdbf90e91888fcded2: Neither blocks nor log segments found. Creating new log.
I20260812 09:46:15.553742 31143 tablet_bootstrap.cc:654] T 64f571599c984eb59ebd74a7c2327a21 P fda0e5942037408185325d3fe00eea11: Neither blocks nor log segments found. Creating new log.
I20260812 09:46:15.554662 31145 tablet_bootstrap.cc:654] T 64f571599c984eb59ebd74a7c2327a21 P f7027607aa404406bd3c9b9e935bc9f7: Neither blocks nor log segments found. Creating new log.
I20260812 09:46:15.554770 31142 log.cc:824] T 64f571599c984eb59ebd74a7c2327a21 P 0e60857e6df448cdbf90e91888fcded2: Log is configured to *not* fsync() on all Append() calls
I20260812 09:46:15.555001 31143 log.cc:824] T 64f571599c984eb59ebd74a7c2327a21 P fda0e5942037408185325d3fe00eea11: Log is configured to *not* fsync() on all Append() calls
I20260812 09:46:15.555913 31145 log.cc:824] T 64f571599c984eb59ebd74a7c2327a21 P f7027607aa404406bd3c9b9e935bc9f7: Log is configured to *not* fsync() on all Append() calls
I20260812 09:46:15.556965 31142 tablet_bootstrap.cc:492] T 64f571599c984eb59ebd74a7c2327a21 P 0e60857e6df448cdbf90e91888fcded2: No bootstrap required, opened a new log
I20260812 09:46:15.557174 31142 ts_tablet_manager.cc:1397] T 64f571599c984eb59ebd74a7c2327a21 P 0e60857e6df448cdbf90e91888fcded2: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260812 09:46:15.557180 31143 tablet_bootstrap.cc:492] T 64f571599c984eb59ebd74a7c2327a21 P fda0e5942037408185325d3fe00eea11: No bootstrap required, opened a new log
I20260812 09:46:15.557371 31143 ts_tablet_manager.cc:1397] T 64f571599c984eb59ebd74a7c2327a21 P fda0e5942037408185325d3fe00eea11: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260812 09:46:15.557866 31145 tablet_bootstrap.cc:492] T 64f571599c984eb59ebd74a7c2327a21 P f7027607aa404406bd3c9b9e935bc9f7: No bootstrap required, opened a new log
I20260812 09:46:15.558036 31145 ts_tablet_manager.cc:1397] T 64f571599c984eb59ebd74a7c2327a21 P f7027607aa404406bd3c9b9e935bc9f7: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260812 09:46:15.560112 31142 raft_consensus.cc:348] T 64f571599c984eb59ebd74a7c2327a21 P 0e60857e6df448cdbf90e91888fcded2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e60857e6df448cdbf90e91888fcded2" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 43747 } } peers { permanent_uuid: "f7027607aa404406bd3c9b9e935bc9f7" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 37439 } } peers { permanent_uuid: "fda0e5942037408185325d3fe00eea11" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 42519 } }
I20260812 09:46:15.560425 31142 raft_consensus.cc:374] T 64f571599c984eb59ebd74a7c2327a21 P 0e60857e6df448cdbf90e91888fcded2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:46:15.560499 31142 raft_consensus.cc:729] T 64f571599c984eb59ebd74a7c2327a21 P 0e60857e6df448cdbf90e91888fcded2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e60857e6df448cdbf90e91888fcded2, State: Initialized, Role: FOLLOWER
I20260812 09:46:15.561050 31142 consensus_queue.cc:260] T 64f571599c984eb59ebd74a7c2327a21 P 0e60857e6df448cdbf90e91888fcded2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e60857e6df448cdbf90e91888fcded2" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 43747 } } peers { permanent_uuid: "f7027607aa404406bd3c9b9e935bc9f7" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 37439 } } peers { permanent_uuid: "fda0e5942037408185325d3fe00eea11" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 42519 } }
I20260812 09:46:15.561607 31143 raft_consensus.cc:348] T 64f571599c984eb59ebd74a7c2327a21 P fda0e5942037408185325d3fe00eea11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e60857e6df448cdbf90e91888fcded2" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 43747 } } peers { permanent_uuid: "f7027607aa404406bd3c9b9e935bc9f7" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 37439 } } peers { permanent_uuid: "fda0e5942037408185325d3fe00eea11" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 42519 } }
I20260812 09:46:15.561666 31145 raft_consensus.cc:348] T 64f571599c984eb59ebd74a7c2327a21 P f7027607aa404406bd3c9b9e935bc9f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e60857e6df448cdbf90e91888fcded2" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 43747 } } peers { permanent_uuid: "f7027607aa404406bd3c9b9e935bc9f7" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 37439 } } peers { permanent_uuid: "fda0e5942037408185325d3fe00eea11" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 42519 } }
I20260812 09:46:15.561869 31143 raft_consensus.cc:374] T 64f571599c984eb59ebd74a7c2327a21 P fda0e5942037408185325d3fe00eea11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:46:15.561926 31145 raft_consensus.cc:374] T 64f571599c984eb59ebd74a7c2327a21 P f7027607aa404406bd3c9b9e935bc9f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:46:15.561923 31143 raft_consensus.cc:729] T 64f571599c984eb59ebd74a7c2327a21 P fda0e5942037408185325d3fe00eea11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fda0e5942037408185325d3fe00eea11, State: Initialized, Role: FOLLOWER
I20260812 09:46:15.561972 31145 raft_consensus.cc:729] T 64f571599c984eb59ebd74a7c2327a21 P f7027607aa404406bd3c9b9e935bc9f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7027607aa404406bd3c9b9e935bc9f7, State: Initialized, Role: FOLLOWER
I20260812 09:46:15.562122 31142 ts_tablet_manager.cc:1428] T 64f571599c984eb59ebd74a7c2327a21 P 0e60857e6df448cdbf90e91888fcded2: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260812 09:46:15.562255 30887 heartbeater.cc:503] Master 127.13.99.62:41023 was elected leader, sending a full tablet report...
I20260812 09:46:15.562398 31143 consensus_queue.cc:260] T 64f571599c984eb59ebd74a7c2327a21 P fda0e5942037408185325d3fe00eea11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e60857e6df448cdbf90e91888fcded2" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 43747 } } peers { permanent_uuid: "f7027607aa404406bd3c9b9e935bc9f7" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 37439 } } peers { permanent_uuid: "fda0e5942037408185325d3fe00eea11" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 42519 } }
I20260812 09:46:15.562398 31145 consensus_queue.cc:260] T 64f571599c984eb59ebd74a7c2327a21 P f7027607aa404406bd3c9b9e935bc9f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e60857e6df448cdbf90e91888fcded2" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 43747 } } peers { permanent_uuid: "f7027607aa404406bd3c9b9e935bc9f7" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 37439 } } peers { permanent_uuid: "fda0e5942037408185325d3fe00eea11" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 42519 } }
I20260812 09:46:15.563333 31143 ts_tablet_manager.cc:1428] T 64f571599c984eb59ebd74a7c2327a21 P fda0e5942037408185325d3fe00eea11: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260812 09:46:15.563333 31145 ts_tablet_manager.cc:1428] T 64f571599c984eb59ebd74a7c2327a21 P f7027607aa404406bd3c9b9e935bc9f7: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260812 09:46:15.563558 31113 heartbeater.cc:503] Master 127.13.99.62:41023 was elected leader, sending a full tablet report...
I20260812 09:46:15.564607 30672 heartbeater.cc:503] Master 127.13.99.62:41023 was elected leader, sending a full tablet report...
I20260812 09:46:15.664983 31152 raft_consensus.cc:482] T 64f571599c984eb59ebd74a7c2327a21 P fda0e5942037408185325d3fe00eea11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 09:46:15.665287 31152 raft_consensus.cc:504] T 64f571599c984eb59ebd74a7c2327a21 P fda0e5942037408185325d3fe00eea11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e60857e6df448cdbf90e91888fcded2" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 43747 } } peers { permanent_uuid: "f7027607aa404406bd3c9b9e935bc9f7" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 37439 } } peers { permanent_uuid: "fda0e5942037408185325d3fe00eea11" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 42519 } }
I20260812 09:46:15.666558 31152 leader_election.cc:288] T 64f571599c984eb59ebd74a7c2327a21 P fda0e5942037408185325d3fe00eea11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e60857e6df448cdbf90e91888fcded2 (127.13.99.2:43747), f7027607aa404406bd3c9b9e935bc9f7 (127.13.99.3:37439)
I20260812 09:46:15.670687 31134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:46:15.666959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.1:43247 (local address 127.13.99.2:43747)
0812 09:46:15.667204 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0812 09:46:15.667210 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 09:46:15.667385 (+ 175us) server_negotiation.cc:407] Connection header received
0812 09:46:15.667565 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:46:15.667568 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:46:15.667607 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:46:15.667689 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:46:15.667985 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:46:15.668554 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:46:15.669713 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:46:15.670112 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:46:15.670229 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:46:15.670297 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0812 09:46:15.670490 (+ 193us) server_negotiation.cc:299] Negotiation successful
0812 09:46:15.670559 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147}
I20260812 09:46:15.670691 31141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:46:15.667039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.1:53899 (local address 127.13.99.3:37439)
0812 09:46:15.667191 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0812 09:46:15.667195 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:46:15.667412 (+ 217us) server_negotiation.cc:407] Connection header received
0812 09:46:15.667576 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:46:15.667579 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:46:15.667631 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:46:15.667714 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:46:15.667987 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:46:15.668549 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:46:15.669705 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:46:15.670121 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:46:15.670239 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:46:15.670304 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0812 09:46:15.670499 (+ 195us) server_negotiation.cc:299] Negotiation successful
0812 09:46:15.670578 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260812 09:46:15.670710 31155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:46:15.666848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.2:43747 (local address 127.13.99.1:43247)
0812 09:46:15.667284 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0812 09:46:15.667296 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 09:46:15.667434 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:46:15.667712 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:46:15.667716 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:46:15.667732 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:46:15.667871 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:46:15.667877 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:46:15.668660 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:46:15.668665 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:46:15.669586 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:46:15.669596 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:46:15.670233 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:46:15.670237 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:46:15.670338 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:46:15.670369 (+ 31us) client_negotiation.cc:769] Sending connection context
0812 09:46:15.670468 (+ 99us) client_negotiation.cc:240] Negotiation successful
0812 09:46:15.670544 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"mutex_wait_us":64,"spinlock_wait_cycles":15232,"thread_start_us":139,"threads_started":1}
I20260812 09:46:15.670714 31154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:46:15.666864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.3:37439 (local address 127.13.99.1:53899)
0812 09:46:15.667318 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0812 09:46:15.667330 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 09:46:15.667461 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:46:15.667728 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:46:15.667731 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:46:15.667747 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:46:15.667874 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:46:15.667879 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:46:15.668665 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:46:15.668668 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:46:15.669577 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:46:15.669587 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:46:15.670245 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:46:15.670248 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:46:15.670372 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:46:15.670391 (+ 19us) client_negotiation.cc:769] Sending connection context
0812 09:46:15.670480 (+ 89us) client_negotiation.cc:240] Negotiation successful
0812 09:46:15.670557 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"mutex_wait_us":42,"thread_start_us":135,"threads_started":1}
I20260812 09:46:15.671386 30811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64f571599c984eb59ebd74a7c2327a21" candidate_uuid: "fda0e5942037408185325d3fe00eea11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e60857e6df448cdbf90e91888fcded2" is_pre_election: true
I20260812 09:46:15.671617 31038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64f571599c984eb59ebd74a7c2327a21" candidate_uuid: "fda0e5942037408185325d3fe00eea11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7027607aa404406bd3c9b9e935bc9f7" is_pre_election: true
I20260812 09:46:15.671725 30811 raft_consensus.cc:2454] T 64f571599c984eb59ebd74a7c2327a21 P 0e60857e6df448cdbf90e91888fcded2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fda0e5942037408185325d3fe00eea11 in term 0.
I20260812 09:46:15.671944 31038 raft_consensus.cc:2454] T 64f571599c984eb59ebd74a7c2327a21 P f7027607aa404406bd3c9b9e935bc9f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fda0e5942037408185325d3fe00eea11 in term 0.
I20260812 09:46:15.672245 30476 leader_election.cc:302] T 64f571599c984eb59ebd74a7c2327a21 P fda0e5942037408185325d3fe00eea11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e60857e6df448cdbf90e91888fcded2, fda0e5942037408185325d3fe00eea11; no voters:
I20260812 09:46:15.672586 31152 raft_consensus.cc:2781] T 64f571599c984eb59ebd74a7c2327a21 P fda0e5942037408185325d3fe00eea11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 09:46:15.672676 31152 raft_consensus.cc:482] T 64f571599c984eb59ebd74a7c2327a21 P fda0e5942037408185325d3fe00eea11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 09:46:15.672708 31152 raft_consensus.cc:3037] T 64f571599c984eb59ebd74a7c2327a21 P fda0e5942037408185325d3fe00eea11 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:46:15.673795 31152 raft_consensus.cc:504] T 64f571599c984eb59ebd74a7c2327a21 P fda0e5942037408185325d3fe00eea11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e60857e6df448cdbf90e91888fcded2" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 43747 } } peers { permanent_uuid: "f7027607aa404406bd3c9b9e935bc9f7" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 37439 } } peers { permanent_uuid: "fda0e5942037408185325d3fe00eea11" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 42519 } }
I20260812 09:46:15.674183 31152 leader_election.cc:288] T 64f571599c984eb59ebd74a7c2327a21 P fda0e5942037408185325d3fe00eea11 [CANDIDATE]: Term 1 election: Requested vote from peers 0e60857e6df448cdbf90e91888fcded2 (127.13.99.2:43747), f7027607aa404406bd3c9b9e935bc9f7 (127.13.99.3:37439)
I20260812 09:46:15.674518 30811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64f571599c984eb59ebd74a7c2327a21" candidate_uuid: "fda0e5942037408185325d3fe00eea11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e60857e6df448cdbf90e91888fcded2"
I20260812 09:46:15.674518 31038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64f571599c984eb59ebd74a7c2327a21" candidate_uuid: "fda0e5942037408185325d3fe00eea11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7027607aa404406bd3c9b9e935bc9f7"
I20260812 09:46:15.674647 31038 raft_consensus.cc:3037] T 64f571599c984eb59ebd74a7c2327a21 P f7027607aa404406bd3c9b9e935bc9f7 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:46:15.674647 30811 raft_consensus.cc:3037] T 64f571599c984eb59ebd74a7c2327a21 P 0e60857e6df448cdbf90e91888fcded2 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:46:15.675994 31038 raft_consensus.cc:2454] T 64f571599c984eb59ebd74a7c2327a21 P f7027607aa404406bd3c9b9e935bc9f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fda0e5942037408185325d3fe00eea11 in term 1.
I20260812 09:46:15.675994 30811 raft_consensus.cc:2454] T 64f571599c984eb59ebd74a7c2327a21 P 0e60857e6df448cdbf90e91888fcded2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fda0e5942037408185325d3fe00eea11 in term 1.
I20260812 09:46:15.676277 30476 leader_election.cc:302] T 64f571599c984eb59ebd74a7c2327a21 P fda0e5942037408185325d3fe00eea11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e60857e6df448cdbf90e91888fcded2, fda0e5942037408185325d3fe00eea11; no voters:
I20260812 09:46:15.676450 31152 raft_consensus.cc:2781] T 64f571599c984eb59ebd74a7c2327a21 P fda0e5942037408185325d3fe00eea11 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 09:46:15.676720 31152 raft_consensus.cc:686] T 64f571599c984eb59ebd74a7c2327a21 P fda0e5942037408185325d3fe00eea11 [term 1 LEADER]: Becoming Leader. State: Replica: fda0e5942037408185325d3fe00eea11, State: Running, Role: LEADER
I20260812 09:46:15.677139 31152 consensus_queue.cc:237] T 64f571599c984eb59ebd74a7c2327a21 P fda0e5942037408185325d3fe00eea11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e60857e6df448cdbf90e91888fcded2" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 43747 } } peers { permanent_uuid: "f7027607aa404406bd3c9b9e935bc9f7" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 37439 } } peers { permanent_uuid: "fda0e5942037408185325d3fe00eea11" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 42519 } }
I20260812 09:46:15.680447 30326 catalog_manager.cc:5495] T 64f571599c984eb59ebd74a7c2327a21 P fda0e5942037408185325d3fe00eea11 reported cstate change: term changed from 0 to 1, leader changed from <none> to fda0e5942037408185325d3fe00eea11 (127.13.99.1). New cstate: current_term: 1 leader_uuid: "fda0e5942037408185325d3fe00eea11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e60857e6df448cdbf90e91888fcded2" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 43747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7027607aa404406bd3c9b9e935bc9f7" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 37439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fda0e5942037408185325d3fe00eea11" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 42519 } health_report { overall_health: HEALTHY } } }
I20260812 09:46:15.729012 31138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:46:15.724692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47118 (local address 127.13.99.1:42519)
0812 09:46:15.724844 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0812 09:46:15.724849 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:46:15.724915 (+ 66us) server_negotiation.cc:407] Connection header received
0812 09:46:15.725129 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:46:15.725133 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:46:15.725216 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:46:15.725343 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 09:46:15.725654 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:46:15.726294 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:46:15.727285 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:46:15.727620 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:46:15.727731 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:46:15.728087 (+ 356us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 09:46:15.728647 (+ 560us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 09:46:15.728773 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0812 09:46:15.728815 (+ 42us) server_negotiation.cc:299] Negotiation successful
0812 09:46:15.728872 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260812 09:46:15.736359 31038 raft_consensus.cc:1261] T 64f571599c984eb59ebd74a7c2327a21 P f7027607aa404406bd3c9b9e935bc9f7 [term 1 FOLLOWER]: Refusing update from remote peer fda0e5942037408185325d3fe00eea11: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 09:46:15.736397 30811 raft_consensus.cc:1261] T 64f571599c984eb59ebd74a7c2327a21 P 0e60857e6df448cdbf90e91888fcded2 [term 1 FOLLOWER]: Refusing update from remote peer fda0e5942037408185325d3fe00eea11: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 09:46:15.737275 31152 consensus_queue.cc:1035] T 64f571599c984eb59ebd74a7c2327a21 P fda0e5942037408185325d3fe00eea11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e60857e6df448cdbf90e91888fcded2" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 43747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 09:46:15.738977 31152 consensus_queue.cc:1035] T 64f571599c984eb59ebd74a7c2327a21 P fda0e5942037408185325d3fe00eea11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7027607aa404406bd3c9b9e935bc9f7" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 09:46:15.740315 31114 tablet.cc:2369] T 64f571599c984eb59ebd74a7c2327a21 P f7027607aa404406bd3c9b9e935bc9f7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 09:46:15.756271 30329 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39948:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 09:46:15.756544 30329 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39948:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 09:46:15.759673 30329 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 60b9bd9e33d44a91bbdb4c13ef55bf42: Sending DeleteTablet for 3 replicas of tablet 64f571599c984eb59ebd74a7c2327a21
I20260812 09:46:15.760454 30783 tablet_service.cc:1514] Processing DeleteTablet for tablet 64f571599c984eb59ebd74a7c2327a21 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 09:46:15 UTC) from {username='kudu', principal='kudu/127.13.99.62@KRBTEST.COM'} at 127.0.0.1:43276
I20260812 09:46:15.760876 30551 tablet_service.cc:1514] Processing DeleteTablet for tablet 64f571599c984eb59ebd74a7c2327a21 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 09:46:15 UTC) from {username='kudu', principal='kudu/127.13.99.62@KRBTEST.COM'} at 127.0.0.1:47108
I20260812 09:46:15.761271 31177 tablet_replica.cc:330] T 64f571599c984eb59ebd74a7c2327a21 P 0e60857e6df448cdbf90e91888fcded2: stopping tablet replica
I20260812 09:46:15.761243 31008 tablet_service.cc:1514] Processing DeleteTablet for tablet 64f571599c984eb59ebd74a7c2327a21 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 09:46:15 UTC) from {username='kudu', principal='kudu/127.13.99.62@KRBTEST.COM'} at 127.0.0.1:45984
I20260812 09:46:15.761616 31177 raft_consensus.cc:2229] T 64f571599c984eb59ebd74a7c2327a21 P 0e60857e6df448cdbf90e91888fcded2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 09:46:15.761837 31179 tablet_replica.cc:330] T 64f571599c984eb59ebd74a7c2327a21 P fda0e5942037408185325d3fe00eea11: stopping tablet replica
I20260812 09:46:15.761961 31177 raft_consensus.cc:2258] T 64f571599c984eb59ebd74a7c2327a21 P 0e60857e6df448cdbf90e91888fcded2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 09:46:15.762040 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 30446
I20260812 09:46:15.762410 31179 raft_consensus.cc:2229] T 64f571599c984eb59ebd74a7c2327a21 P fda0e5942037408185325d3fe00eea11 [term 1 LEADER]: Raft consensus shutting down.
I20260812 09:46:15.766764 31177 ts_tablet_manager.cc:1905] T 64f571599c984eb59ebd74a7c2327a21 P 0e60857e6df448cdbf90e91888fcded2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 09:46:15.770642 31177 ts_tablet_manager.cc:1918] T 64f571599c984eb59ebd74a7c2327a21 P 0e60857e6df448cdbf90e91888fcded2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 09:46:15.770790 31177 log.cc:1196] T 64f571599c984eb59ebd74a7c2327a21 P 0e60857e6df448cdbf90e91888fcded2: Deleting WAL directory at /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/minicluster-data/ts-1/wal/wals/64f571599c984eb59ebd74a7c2327a21
I20260812 09:46:15.771252 31177 ts_tablet_manager.cc:1939] T 64f571599c984eb59ebd74a7c2327a21 P 0e60857e6df448cdbf90e91888fcded2: Deleting consensus metadata
I20260812 09:46:15.772845 30302 catalog_manager.cc:4841] TS 0e60857e6df448cdbf90e91888fcded2 (127.13.99.2:43747): tablet 64f571599c984eb59ebd74a7c2327a21 (table test-table [id=5e42e2f835314c6493b7f9a041ceda86]) successfully deleted
I20260812 09:46:15.780912 31180 tablet_replica.cc:330] T 64f571599c984eb59ebd74a7c2327a21 P f7027607aa404406bd3c9b9e935bc9f7: stopping tablet replica
I20260812 09:46:15.781411 31180 raft_consensus.cc:2229] T 64f571599c984eb59ebd74a7c2327a21 P f7027607aa404406bd3c9b9e935bc9f7 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260812 09:46:15.781769 30302 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.99.1:42519 (error 108)
I20260812 09:46:15.781833 31180 raft_consensus.cc:2258] T 64f571599c984eb59ebd74a7c2327a21 P f7027607aa404406bd3c9b9e935bc9f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 09:46:15.782081 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 30681
I20260812 09:46:15.801688 31133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:46:15.801244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.1:42519 (local address 127.0.0.1:47120)
0812 09:46:15.801473 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0812 09:46:15.801567 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.1:42519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128}
W20260812 09:46:15.801921 30302 catalog_manager.cc:4584] TS fda0e5942037408185325d3fe00eea11 (127.13.99.1:42519): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 64f571599c984eb59ebd74a7c2327a21: Network error: Client connection negotiation failed: client connection to 127.13.99.1:42519: connect: Connection refused (error 111)
I20260812 09:46:15.804771 31180 ts_tablet_manager.cc:1905] T 64f571599c984eb59ebd74a7c2327a21 P f7027607aa404406bd3c9b9e935bc9f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 09:46:15.807901 31180 ts_tablet_manager.cc:1918] T 64f571599c984eb59ebd74a7c2327a21 P f7027607aa404406bd3c9b9e935bc9f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 09:46:15.808005 31180 log.cc:1196] T 64f571599c984eb59ebd74a7c2327a21 P f7027607aa404406bd3c9b9e935bc9f7: Deleting WAL directory at /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786527905902985-13708-0/minicluster-data/ts-2/wal/wals/64f571599c984eb59ebd74a7c2327a21
I20260812 09:46:15.808334 31180 ts_tablet_manager.cc:1939] T 64f571599c984eb59ebd74a7c2327a21 P f7027607aa404406bd3c9b9e935bc9f7: Deleting consensus metadata
I20260812 09:46:15.809631 30303 catalog_manager.cc:4841] TS f7027607aa404406bd3c9b9e935bc9f7 (127.13.99.3:37439): tablet 64f571599c984eb59ebd74a7c2327a21 (table test-table [id=5e42e2f835314c6493b7f9a041ceda86]) successfully deleted
I20260812 09:46:15.813531 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 30898
I20260812 09:46:15.833700 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 30270
2026-08-12T09:46:15Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2737 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 09:46:15 dist-test-slave-jq4t krb5kdc[31187](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 09:46:15 dist-test-slave-jq4t krb5kdc[31187](info): set up 2 sockets
Aug 12 09:46:15 dist-test-slave-jq4t krb5kdc[31187](info): commencing operation
krb5kdc: starting...
W20260812 09:46:17.112835 13708 mini_kdc.cc:121] Time spent starting KDC: real 1.252s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/krb5kdc/test-user.keytab.
Aug 12 09:46:17 dist-test-slave-jq4t krb5kdc[31187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786527977, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T09:46:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T09:46:17Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.13.99.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.13.99.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.13.99.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/krb5kdc/rangeradmin_127.13.99.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.13.99.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/krb5kdc/rangeradmin_127.13.99.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.13.99.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.13.99.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.13.99.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/krb5kdc/rangerlookup_127.13.99.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.13.99.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/krb5kdc/rangerlookup_127.13.99.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.13.99.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.99.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.99.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/krb5kdc/HTTP_127.13.99.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.13.99.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/krb5kdc/HTTP_127.13.99.20@KRBTEST.COM.keytab.
I20260812 09:46:17.282610 13708 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskJsh0qK/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/minicluster-data/postgres -l logfile start
2026-08-12 09:46:19.839 UTC [31231] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-12 09:46:19.840 UTC [31231] LOG: listening on IPv4 address "127.13.99.20", port 42565
2026-08-12 09:46:19.843 UTC [31231] LOG: listening on Unix socket "/tmp/.s.PGSQL.42565"
2026-08-12 09:46:19.880 UTC [31234] LOG: database system was shut down at 2026-08-12 09:46:18 UTC
2026-08-12 09:46:19.884 UTC [31231] LOG: database system is ready to accept connections
I20260812 09:46:21.061477 13708 mini_postgres.cc:96] Postgres bound to 42565
2026-08-12 09:46:21.070 UTC [31247] FATAL: database "slave" does not exist
127.13.99.20:42565 - accepting connections
I20260812 09:46:21.071584 13708 mini_ranger.cc:161] Starting Ranger...
I20260812 09:46:21.088348 13708 mini_ranger.cc:84] Created miniranger Postgres user
I20260812 09:46:21.599972 13708 mini_ranger.cc:87] Created ranger Postgres database
I20260812 09:46:21.600095 13708 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/minicluster-data/ranger-admin
2026-08-12 09:46:21,934 [I] DB FLAVOR :POSTGRES
2026-08-12 09:46:21,935 [I] --------- Verifying Ranger DB connection ---------
2026-08-12 09:46:21,935 [I] Checking connection..
2026-08-12 09:46:21,935 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJsh0qK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJsh0qK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.99.20:42565/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-12 09:46:22,202 [I] Checking connection passed.
2026-08-12 09:46:22,203 [I] --------- Verifying version history table ---------
2026-08-12 09:46:22,203 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJsh0qK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJsh0qK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.99.20:42565/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 09:46:22,456 [I] Table x_db_version_h does not exist in database ranger
2026-08-12 09:46:22,457 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJsh0qK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJsh0qK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.99.20:42565/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 09:46:22,710 [I] Table x_db_version_h does not exist in database ranger
2026-08-12 09:46:22,710 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-12 09:46:22,710 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJsh0qK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJsh0qK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.99.20:42565/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskJsh0qK/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-12 09:46:22.940 UTC [31357] WARNING: there is no transaction in progress
2026-08-12 09:46:22,950 [I] create_dbversion_catalog.sql file imported successfully
2026-08-12 09:46:22,950 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJsh0qK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJsh0qK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.99.20:42565/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 09:46:23,197 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-12 09:46:23,198 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-12 09:46:23,198 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJsh0qK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJsh0qK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.99.20:42565/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-12 09:46:23,463 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJsh0qK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJsh0qK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.99.20:42565/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-12 09:46:23,712 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJsh0qK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJsh0qK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.99.20:42565/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-jq4t.c.gcp-upstream.internal','N') ;"
2026-08-12 09:46:23,958 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJsh0qK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJsh0qK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.99.20:42565/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-12 09:46:24,221 [I] Table x_portal_user does not exist in database ranger
2026-08-12 09:46:24,221 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJsh0qK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJsh0qK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.99.20:42565/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-12 09:46:24,472 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-12 09:46:24,473 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJsh0qK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJsh0qK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.99.20:42565/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-12 09:46:24,740 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJsh0qK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJsh0qK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.99.20:42565/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-12 09:46:25,007 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-12 09:46:25,007 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJsh0qK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJsh0qK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.99.20:42565/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskJsh0qK/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-12 09:46:26.012 UTC [31548] WARNING: there is no transaction in progress
2026-08-12 09:46:26.029 UTC [31548] WARNING: there is no transaction in progress
2026-08-12 09:46:26.046 UTC [31548] WARNING: there is no transaction in progress
2026-08-12 09:46:26.062 UTC [31548] WARNING: there is no transaction in progress
2026-08-12 09:46:26.077 UTC [31548] WARNING: there is no transaction in progress
2026-08-12 09:46:26.093 UTC [31548] WARNING: there is no transaction in progress
2026-08-12 09:46:26.145 UTC [31548] WARNING: there is no transaction in progress
2026-08-12 09:46:26.151 UTC [31548] WARNING: there is no transaction in progress
2026-08-12 09:46:26.165 UTC [31548] WARNING: there is no transaction in progress
2026-08-12 09:46:26.175 UTC [31548] WARNING: there is no transaction in progress
2026-08-12 09:46:26.186 UTC [31548] WARNING: there is no transaction in progress
2026-08-12 09:46:26.200 UTC [31548] WARNING: there is no transaction in progress
2026-08-12 09:46:26.209 UTC [31548] WARNING: there is no transaction in progress
2026-08-12 09:46:26.216 UTC [31548] WARNING: there is no transaction in progress
2026-08-12 09:46:26.223 UTC [31548] WARNING: there is no transaction in progress
2026-08-12 09:46:26.229 UTC [31548] WARNING: there is no transaction in progress
2026-08-12 09:46:26,906 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-12 09:46:26,907 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJsh0qK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJsh0qK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.99.20:42565/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-12 09:46:27,162 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-12 09:46:27,162 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJsh0qK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJsh0qK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.99.20:42565/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-12 09:46:27,410 [I] Table x_portal_user already exists in database 'ranger'
2026-08-12 09:46:27,410 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJsh0qK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJsh0qK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.99.20:42565/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-12 09:46:27,680 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-12 09:46:27,681 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJsh0qK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJsh0qK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.99.20:42565/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-12 09:46:27,939 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJsh0qK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJsh0qK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.99.20:42565/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-12 09:46:28,187 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJsh0qK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJsh0qK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.99.20:42565/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-jq4t.c.gcp-upstream.internal';"
2026-08-12 09:46:28,443 [I] CORE_DB_SCHEMA import status has been updated
2026-08-12 09:46:28,444 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJsh0qK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJsh0qK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.99.20:42565/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-12 09:46:28,705 [I] DB_PATCHES have already been applied
I20260812 09:46:28.711272 13708 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/minicluster-data/ranger-admin:/tmp/dist-test-taskJsh0qK/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskJsh0qK/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskJsh0qK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJsh0qK/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260812 09:46:28.711360 13708 mini_ranger.cc:193] Using host: 127.13.99.20
I20260812 09:46:28.716336 13708 mini_ranger.cc:227] Ranger admin URL: http://127.13.99.20:40365
Aug 12, 2026 9:46:29 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 12, 2026 9:46:29 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 12, 2026 9:46:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/minicluster-data/ranger-admin/ews/webapp
Aug 12, 2026 9:46:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 12, 2026 9:46:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 12, 2026 9:46:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/minicluster-data/ranger-admin/ews/webapp].
Aug 12, 2026 9:46:29 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260812 09:46:30.106787 13708 mini_ranger.cc:160] Time spent starting Ranger: real 9.035s user 0.005s sys 0.009s
Aug 12, 2026 9:46:49 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 12, 2026 9:46:49 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.MetricsREST
Aug 12, 2026 9:46:49 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 12, 2026 9:46:49 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 12, 2026 9:46:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 12, 2026 9:46:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 12, 2026 9:46:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 12, 2026 9:46:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 12, 2026 9:46:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 12, 2026 9:46:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 12, 2026 9:46:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 12, 2026 9:46:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 12, 2026 9:46:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 12, 2026 9:46:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 12, 2026 9:46:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 12, 2026 9:46:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 12, 2026 9:46:49 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 12, 2026 9:46:50 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260812 09:46:51.037271 13708 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.13.99.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.13.99.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.13.99.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/krb5kdc/rangerkms_127.13.99.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.13.99.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/krb5kdc/rangerkms_127.13.99.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.13.99.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.13.99.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.99.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/krb5kdc/HTTP_127.13.99.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.13.99.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/krb5kdc/HTTP_127.13.99.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 12 09:46:51 dist-test-slave-jq4t krb5kdc[31187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528011, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260812 09:46:51.151130 13708 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260812 09:46:51.161983 13708 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260812 09:46:52.096336 13708 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260812 09:46:52.096438 13708 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/minicluster-data/ranger-kms
I20260812 09:46:52.096565 13708 mini_ranger_kms.cc:227] Using postgres at 127.13.99.20:42565
2026-08-12 09:46:52,482 [I] DB FLAVOR :POSTGRES
2026-08-12 09:46:52,482 [I] --------- Verifying Ranger DB connection ---------
2026-08-12 09:46:52,482 [I] Checking connection
2026-08-12 09:46:52,482 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJsh0qK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.99.20:42565/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-12 09:46:52,757 [I] connection success
2026-08-12 09:46:52,757 [I] --------- Verifying Ranger DB tables ---------
2026-08-12 09:46:52,757 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-12 09:46:52,757 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJsh0qK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.99.20:42565/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-12 09:46:53,075 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-12 09:46:53,076 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-12 09:46:53,076 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-12 09:46:53,076 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJsh0qK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.99.20:42565/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-12 09:46:53,366 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260812 09:46:53.840401 13708 mini_ranger_kms.cc:310] Created kms service
I20260812 09:46:54.054050 13708 mini_ranger_kms.cc:326] Created kudu user
I20260812 09:46:54.099944 13708 mini_ranger_kms.cc:343] Created rangerkms user
Aug 12 09:46:54 dist-test-slave-jq4t krb5kdc[31187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528011, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.13.99.20@KRBTEST.COM
I20260812 09:46:54.496098 13708 mini_ranger_kms.cc:384] Added ranger policy
I20260812 09:46:54.496447 13708 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/minicluster-data/ranger-kms:/tmp/dist-test-taskJsh0qK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJsh0qK/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskJsh0qK/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260812 09:46:54.496469 13708 mini_ranger_kms.cc:240] Using host: 127.13.99.20
I20260812 09:46:54.501595 13708 mini_ranger_kms.cc:276] Ranger KMS PID: 31836
I20260812 09:46:54.501748 13708 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.13.99.20:51121
Aug 12, 2026 9:46:55 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 12, 2026 9:46:55 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 12, 2026 9:46:55 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 12, 2026 9:46:55 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 12, 2026 9:46:56 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/minicluster-data/ranger-kms/ews/webapp].
Aug 12, 2026 9:46:56 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 12 09:46:56 dist-test-slave-jq4t krb5kdc[31187](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786528016, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.13.99.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:46:57.382534 13708 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.231s user 0.004s sys 0.362s
I20260812 09:46:57.382664 13708 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260812 09:46:57.382769 13708 mini_ranger_kms.cc:401] 127.13.99.20:51121/kms/v1/keys
Aug 12 09:46:57 dist-test-slave-jq4t krb5kdc[31187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528016, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.13.99.20@KRBTEST.COM for HTTP/127.13.99.20@KRBTEST.COM
Aug 12 09:46:57 dist-test-slave-jq4t krb5kdc[31187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528016, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.13.99.20@KRBTEST.COM for HTTP/127.13.99.20@KRBTEST.COM
Aug 12 09:46:59 dist-test-slave-jq4t krb5kdc[31187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528019, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.13.99.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.99.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.99.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab.
Entry for principal kudu/127.13.99.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab.
I20260812 09:46:59.956598 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.62:45007
--webserver_interface=127.13.99.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab
--principal=kudu/127.13.99.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:39543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.99.62:45007
--ranger_config_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.99.20:51121/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:47:00.080673 31922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:47:00.080951 31922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:47:00.080993 31922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:47:00.085173 31922 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 09:47:00.085243 31922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:47:00.085264 31922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:47:00.085283 31922 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 09:47:00.085299 31922 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 09:47:00.091256 31922 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:39543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/minicluster-data/master-0/wal
--ranger_kms_url=127.13.99.20:51121/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.13.99.62:45007
--ranger_config_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab
--principal=kudu/127.13.99.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.62:45007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.99.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31922
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:47:00.092363 31922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:47:00.093533 31922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:47:00.103001 31933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:47:00.103060 31922 server_base.cc:1034] running on GCE node
W20260812 09:47:00.102982 31932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:47:00.102988 31937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:47:00.103957 31922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:47:00.105120 31922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:47:00.106312 31922 hybrid_clock.cc:648] HybridClock initialized: now 1786528020106290 us; error 32 us; skew 500 ppm
Aug 12 09:47:00 dist-test-slave-jq4t krb5kdc[31187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528020, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:47:00.109207 31922 init.cc:377] Logged in from keytab as kudu/127.13.99.62@KRBTEST.COM (short username kudu)
I20260812 09:47:00.110056 31922 webserver.cc:460] Webserver started at http://127.13.99.62:40509/ using document root <none> and password file <none>
I20260812 09:47:00.110581 31922 fs_manager.cc:359] Metadata directory not provided
I20260812 09:47:00.110636 31922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:47:00.110819 31922 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 09:47:00 dist-test-slave-jq4t krb5kdc[31187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528020, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.62@KRBTEST.COM for HTTP/127.13.99.20@KRBTEST.COM
I20260812 09:47:00.175726 31922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/minicluster-data/master-0/data/instance:
uuid: "2e4eae9476df441e94bd11701ebb9cf8"
format_stamp: "Formatted at 2026-08-12 09:47:00 on dist-test-slave-jq4t"
server_key: "fdb053172d3e6c499016662f78c35650"
server_key_iv: "da25471b5b58a7c3ff73af62e062f2f5"
server_key_version: "kuduclusterkey@0"
I20260812 09:47:00.176394 31922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/minicluster-data/master-0/wal/instance:
uuid: "2e4eae9476df441e94bd11701ebb9cf8"
format_stamp: "Formatted at 2026-08-12 09:47:00 on dist-test-slave-jq4t"
server_key: "fdb053172d3e6c499016662f78c35650"
server_key_iv: "da25471b5b58a7c3ff73af62e062f2f5"
server_key_version: "kuduclusterkey@0"
I20260812 09:47:00.180425 31922 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260812 09:47:00.194409 31945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:47:00.195962 31922 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 09:47:00.196084 31922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/minicluster-data/master-0/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/minicluster-data/master-0/wal
uuid: "2e4eae9476df441e94bd11701ebb9cf8"
format_stamp: "Formatted at 2026-08-12 09:47:00 on dist-test-slave-jq4t"
server_key: "fdb053172d3e6c499016662f78c35650"
server_key_iv: "da25471b5b58a7c3ff73af62e062f2f5"
server_key_version: "kuduclusterkey@0"
I20260812 09:47:00.196245 31922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:47:00.226629 31922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:47:00.229988 31922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:47:00.230201 31922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:47:00.239464 31922 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.62:45007
I20260812 09:47:00.239550 32033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.62:45007 every 8 connection(s)
I20260812 09:47:00.240541 31922 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
I20260812 09:47:00.243489 32034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:47:00.245736 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 31922
I20260812 09:47:00.245862 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/minicluster-data/master-0/wal/instance
I20260812 09:47:00.246187 13708 external_mini_cluster.cc:1428] Setting key d79a793d07144663ba3c4c0552e97c7a
I20260812 09:47:00.250423 32034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e4eae9476df441e94bd11701ebb9cf8: Bootstrap starting.
I20260812 09:47:00.253273 32034 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e4eae9476df441e94bd11701ebb9cf8: Neither blocks nor log segments found. Creating new log.
I20260812 09:47:00.254207 32034 log.cc:824] T 00000000000000000000000000000000 P 2e4eae9476df441e94bd11701ebb9cf8: Log is configured to *not* fsync() on all Append() calls
I20260812 09:47:00.256699 32034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e4eae9476df441e94bd11701ebb9cf8: No bootstrap required, opened a new log
Aug 12 09:47:00 dist-test-slave-jq4t krb5kdc[31187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528019, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
I20260812 09:47:00.261551 32034 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e4eae9476df441e94bd11701ebb9cf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e4eae9476df441e94bd11701ebb9cf8" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 45007 } }
I20260812 09:47:00.261821 32034 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2e4eae9476df441e94bd11701ebb9cf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:47:00.261965 32034 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e4eae9476df441e94bd11701ebb9cf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e4eae9476df441e94bd11701ebb9cf8, State: Initialized, Role: FOLLOWER
I20260812 09:47:00.262497 32034 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e4eae9476df441e94bd11701ebb9cf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e4eae9476df441e94bd11701ebb9cf8" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 45007 } }
I20260812 09:47:00.262660 32034 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e4eae9476df441e94bd11701ebb9cf8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 09:47:00.262725 32034 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e4eae9476df441e94bd11701ebb9cf8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 09:47:00.262820 32034 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e4eae9476df441e94bd11701ebb9cf8 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:47:00.263590 32041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:47:00.247751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38010 (local address 127.13.99.62:45007)
0812 09:47:00.248284 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0812 09:47:00.248304 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0812 09:47:00.248350 (+ 46us) server_negotiation.cc:407] Connection header received
0812 09:47:00.249283 (+ 933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:47:00.249307 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:47:00.249712 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:47:00.250071 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:47:00.251023 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:47:00.252663 (+ 1640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:47:00.253938 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:47:00.254647 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:47:00.254834 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:47:00.258057 (+ 3223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:47:00.258064 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:47:00.258087 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:47:00.258127 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:47:00.260352 (+ 2225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:47:00.260853 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:47:00.260860 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:47:00.260868 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:47:00.261012 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:47:00.261283 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:47:00.261286 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:47:00.261288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:47:00.261797 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:47:00.262015 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0812 09:47:00.262272 (+ 257us) server_negotiation.cc:299] Negotiation successful
0812 09:47:00.262556 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":147,"threads_started":1}
I20260812 09:47:00.264060 32034 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e4eae9476df441e94bd11701ebb9cf8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e4eae9476df441e94bd11701ebb9cf8" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 45007 } }
I20260812 09:47:00.264412 32034 leader_election.cc:302] T 00000000000000000000000000000000 P 2e4eae9476df441e94bd11701ebb9cf8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e4eae9476df441e94bd11701ebb9cf8; no voters:
I20260812 09:47:00.264803 32034 leader_election.cc:288] T 00000000000000000000000000000000 P 2e4eae9476df441e94bd11701ebb9cf8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 09:47:00.264861 32045 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e4eae9476df441e94bd11701ebb9cf8 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 09:47:00.265169 32045 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e4eae9476df441e94bd11701ebb9cf8 [term 1 LEADER]: Becoming Leader. State: Replica: 2e4eae9476df441e94bd11701ebb9cf8, State: Running, Role: LEADER
I20260812 09:47:00.265537 32045 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e4eae9476df441e94bd11701ebb9cf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e4eae9476df441e94bd11701ebb9cf8" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 45007 } }
I20260812 09:47:00.266096 32034 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e4eae9476df441e94bd11701ebb9cf8 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 09:47:00.267400 32048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e4eae9476df441e94bd11701ebb9cf8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2e4eae9476df441e94bd11701ebb9cf8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e4eae9476df441e94bd11701ebb9cf8" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 45007 } } }
I20260812 09:47:00.267531 32048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e4eae9476df441e94bd11701ebb9cf8 [sys.catalog]: This master's current role is: LEADER
I20260812 09:47:00.267802 32049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e4eae9476df441e94bd11701ebb9cf8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e4eae9476df441e94bd11701ebb9cf8. Latest consensus state: current_term: 1 leader_uuid: "2e4eae9476df441e94bd11701ebb9cf8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e4eae9476df441e94bd11701ebb9cf8" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 45007 } } }
I20260812 09:47:00.267906 32049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e4eae9476df441e94bd11701ebb9cf8 [sys.catalog]: This master's current role is: LEADER
I20260812 09:47:00.268414 32065 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 09:47:00.269469 32034 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260812 09:47:00.274917 32065 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 09:47:00.283025 32065 catalog_manager.cc:1347] Generated new cluster ID: 7d5305d4a32a439bbfd5f90bae6e4701
I20260812 09:47:00.283140 32065 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 09:47:00.298156 32065 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 09:47:00.301101 32065 catalog_manager.cc:1504] Loading token signing keys...
I20260812 09:47:00.308039 32065 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2e4eae9476df441e94bd11701ebb9cf8: Generated new TSK 0
I20260812 09:47:00.308789 32065 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.99.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.99.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.99.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.1.keytab.
Entry for principal kudu/127.13.99.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.1.keytab.
I20260812 09:47:01.051297 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.99.20:51121/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.99.1:0
--local_ip_for_outbound_sockets=127.13.99.1
--webserver_interface=127.13.99.1
--webserver_port=0
--tserver_master_addrs=127.13.99.62:45007
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.1.keytab
--principal=kudu/127.13.99.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:39543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:47:01.224145 32118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:47:01.224509 32118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:47:01.224587 32118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:47:01.230702 32118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:47:01.230820 32118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:47:01.230929 32118 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.99.1
I20260812 09:47:01.238777 32118 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:39543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.13.99.20:51121/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.1.keytab
--principal=kudu/127.13.99.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.99.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.99.1
--webserver_port=0
--tserver_master_addrs=127.13.99.62:45007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32118
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.99.1
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:47:01.240173 32118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:47:01.241616 32118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:47:01.254665 32133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:47:01.254680 32134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:47:01.254611 32137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:47:01.256064 32118 server_base.cc:1034] running on GCE node
I20260812 09:47:01.256644 32118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:47:01.257516 32118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:47:01.258730 32118 hybrid_clock.cc:648] HybridClock initialized: now 1786528021258711 us; error 31 us; skew 500 ppm
Aug 12 09:47:01 dist-test-slave-jq4t krb5kdc[31187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528021, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:47:01.265614 32118 init.cc:377] Logged in from keytab as kudu/127.13.99.1@KRBTEST.COM (short username kudu)
I20260812 09:47:01.267102 32118 webserver.cc:460] Webserver started at http://127.13.99.1:46415/ using document root <none> and password file <none>
I20260812 09:47:01.267930 32118 fs_manager.cc:359] Metadata directory not provided
I20260812 09:47:01.268038 32118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:47:01.268355 32118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 09:47:01 dist-test-slave-jq4t krb5kdc[31187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528021, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.1@KRBTEST.COM for HTTP/127.13.99.20@KRBTEST.COM
I20260812 09:47:01.285252 32118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/minicluster-data/ts-0/data/instance:
uuid: "efbbdd7325f740b3abc76c509ba76e4b"
format_stamp: "Formatted at 2026-08-12 09:47:01 on dist-test-slave-jq4t"
server_key: "bfe6e53913146b710017d3d2f36935ba"
server_key_iv: "04bf8ec70f25dd1513a07d86e5a92ae3"
server_key_version: "kuduclusterkey@0"
I20260812 09:47:01.285992 32118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/minicluster-data/ts-0/wal/instance:
uuid: "efbbdd7325f740b3abc76c509ba76e4b"
format_stamp: "Formatted at 2026-08-12 09:47:01 on dist-test-slave-jq4t"
server_key: "bfe6e53913146b710017d3d2f36935ba"
server_key_iv: "04bf8ec70f25dd1513a07d86e5a92ae3"
server_key_version: "kuduclusterkey@0"
I20260812 09:47:01.291326 32118 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 09:47:01.310856 32149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:47:01.312430 32118 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260812 09:47:01.312614 32118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/minicluster-data/ts-0/wal
uuid: "efbbdd7325f740b3abc76c509ba76e4b"
format_stamp: "Formatted at 2026-08-12 09:47:01 on dist-test-slave-jq4t"
server_key: "bfe6e53913146b710017d3d2f36935ba"
server_key_iv: "04bf8ec70f25dd1513a07d86e5a92ae3"
server_key_version: "kuduclusterkey@0"
I20260812 09:47:01.312767 32118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:47:01.336880 32118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:47:01.340755 32118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:47:01.341017 32118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:47:01.341780 32118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 09:47:01.342932 32118 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:47:01.343046 32118 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 09:47:01.343176 32118 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:47:01.343259 32118 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 09:47:01.357048 32118 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.1:38159
I20260812 09:47:01.357087 32361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.1:38159 every 8 connection(s)
I20260812 09:47:01.358167 32118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/minicluster-data/ts-0/data/info.pb
I20260812 09:47:01.361402 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 32118
I20260812 09:47:01.361529 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/minicluster-data/ts-0/wal/instance
I20260812 09:47:01.361907 13708 external_mini_cluster.cc:1428] Setting key 95cccf13393e415b2a3df9f8d9431f90
Aug 12 09:47:01 dist-test-slave-jq4t krb5kdc[31187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528021, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.1@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.99.2@KRBTEST.COM; defaulting to no policy
I20260812 09:47:01.380460 32371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:47:01.360423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.1:60663 (local address 127.13.99.62:45007)
0812 09:47:01.360760 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0812 09:47:01.360764 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:47:01.365780 (+ 5016us) server_negotiation.cc:407] Connection header received
0812 09:47:01.367275 (+ 1495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:47:01.367281 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:47:01.367349 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:47:01.367422 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:47:01.369248 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:47:01.370171 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:47:01.371963 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:47:01.372408 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:47:01.372579 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:47:01.376025 (+ 3446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:47:01.376031 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:47:01.376037 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:47:01.376074 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:47:01.377841 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:47:01.378380 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:47:01.378386 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:47:01.378391 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:47:01.378491 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:47:01.378866 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:47:01.378870 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:47:01.378872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:47:01.379059 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:47:01.379173 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0812 09:47:01.380024 (+ 851us) server_negotiation.cc:299] Negotiation successful
0812 09:47:01.380182 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":145,"threads_started":1}
I20260812 09:47:01.382274 32373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:47:01.361024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:45007 (local address 127.13.99.1:60663)
0812 09:47:01.363393 (+ 2369us) negotiation.cc:107] Waiting for socket to connect
0812 09:47:01.363457 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0812 09:47:01.366992 (+ 3535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:47:01.367902 (+ 910us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:47:01.367910 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:47:01.368197 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:47:01.369056 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:47:01.369083 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:47:01.370371 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:47:01.370377 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:47:01.371829 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:47:01.371841 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:47:01.372803 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:47:01.372808 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:47:01.373063 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:47:01.373929 (+ 866us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:47:01.373952 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:47:01.375793 (+ 1841us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:47:01.377978 (+ 2185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:47:01.377984 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:47:01.378000 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:47:01.378259 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:47:01.378611 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:47:01.378613 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:47:01.378615 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:47:01.378750 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:47:01.379317 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:47:01.379325 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:47:01.379695 (+ 370us) client_negotiation.cc:769] Sending connection context
0812 09:47:01.380680 (+ 985us) client_negotiation.cc:240] Negotiation successful
0812 09:47:01.381111 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1608,"thread_start_us":163,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.99.2@KRBTEST.COM" created.
I20260812 09:47:01.389343 32364 heartbeater.cc:347] Connected to a master server at 127.13.99.62:45007
I20260812 09:47:01.389638 32364 heartbeater.cc:464] Registering TS with master...
I20260812 09:47:01.390272 32364 heartbeater.cc:511] Master 127.13.99.62:45007 requested a full tablet report, sending...
I20260812 09:47:01.392163 31977 ts_manager.cc:194] Registered new tserver with Master: efbbdd7325f740b3abc76c509ba76e4b (127.13.99.1:38159)
I20260812 09:47:01.393440 31977 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.99.1@KRBTEST.COM'} at 127.13.99.1:60663
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.99.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.2.keytab.
Entry for principal kudu/127.13.99.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.2.keytab.
I20260812 09:47:01.413755 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.99.20:51121/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.99.2:0
--local_ip_for_outbound_sockets=127.13.99.2
--webserver_interface=127.13.99.2
--webserver_port=0
--tserver_master_addrs=127.13.99.62:45007
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.2.keytab
--principal=kudu/127.13.99.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:39543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 09:47:01 dist-test-slave-jq4t krb5kdc[31187](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786528021, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260812 09:47:01.553489 32377 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:47:01.553759 32377 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:47:01.553845 32377 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:47:01.557935 32377 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:47:01.558013 32377 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:47:01.558115 32377 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.99.2
I20260812 09:47:01.564687 32377 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:39543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.13.99.20:51121/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.2.keytab
--principal=kudu/127.13.99.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.99.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.99.2
--webserver_port=0
--tserver_master_addrs=127.13.99.62:45007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32377
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.99.2
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:47:01.566037 32377 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:47:01.567400 32377 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 09:47:01.578027 32377 server_base.cc:1034] running on GCE node
W20260812 09:47:01.578341 32390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:47:01.578034 32387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:47:01.578006 32385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:47:01.579103 32377 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:47:01.579735 32377 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:47:01.580924 32377 hybrid_clock.cc:648] HybridClock initialized: now 1786528021580894 us; error 44 us; skew 500 ppm
Aug 12 09:47:01 dist-test-slave-jq4t krb5kdc[31187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528021, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:47:01.584062 32377 init.cc:377] Logged in from keytab as kudu/127.13.99.2@KRBTEST.COM (short username kudu)
I20260812 09:47:01.585103 32377 webserver.cc:460] Webserver started at http://127.13.99.2:39069/ using document root <none> and password file <none>
I20260812 09:47:01.585690 32377 fs_manager.cc:359] Metadata directory not provided
I20260812 09:47:01.585785 32377 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:47:01.586004 32377 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 09:47:01 dist-test-slave-jq4t krb5kdc[31187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528021, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.2@KRBTEST.COM for HTTP/127.13.99.20@KRBTEST.COM
I20260812 09:47:01.599843 32377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/minicluster-data/ts-1/data/instance:
uuid: "bbbc574ad5fa4517b51c85f92b9fe117"
format_stamp: "Formatted at 2026-08-12 09:47:01 on dist-test-slave-jq4t"
server_key: "c79b7b559c812729ab27c5dd7de4c6ec"
server_key_iv: "23b00269f8f5e85ab6f13e43ba201604"
server_key_version: "kuduclusterkey@0"
I20260812 09:47:01.600673 32377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/minicluster-data/ts-1/wal/instance:
uuid: "bbbc574ad5fa4517b51c85f92b9fe117"
format_stamp: "Formatted at 2026-08-12 09:47:01 on dist-test-slave-jq4t"
server_key: "c79b7b559c812729ab27c5dd7de4c6ec"
server_key_iv: "23b00269f8f5e85ab6f13e43ba201604"
server_key_version: "kuduclusterkey@0"
I20260812 09:47:01.608438 32377 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.000s
I20260812 09:47:01.625188 32399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:47:01.626636 32377 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 09:47:01.626785 32377 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/minicluster-data/ts-1/wal
uuid: "bbbc574ad5fa4517b51c85f92b9fe117"
format_stamp: "Formatted at 2026-08-12 09:47:01 on dist-test-slave-jq4t"
server_key: "c79b7b559c812729ab27c5dd7de4c6ec"
server_key_iv: "23b00269f8f5e85ab6f13e43ba201604"
server_key_version: "kuduclusterkey@0"
I20260812 09:47:01.626955 32377 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:47:01.649646 32377 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:47:01.653436 32377 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:47:01.653642 32377 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:47:01.654382 32377 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 09:47:01.655467 32377 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:47:01.655562 32377 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 09:47:01.655634 32377 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:47:01.655676 32377 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 09:47:01.667814 32377 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.2:45283
I20260812 09:47:01.669140 32377 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/minicluster-data/ts-1/data/info.pb
I20260812 09:47:01.677315 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 32377
I20260812 09:47:01.677455 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/minicluster-data/ts-1/wal/instance
I20260812 09:47:01.677785 13708 external_mini_cluster.cc:1428] Setting key edb1517fb6ab0d03810deff757ceecc6
I20260812 09:47:01.682071 32607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.2:45283 every 8 connection(s)
Aug 12 09:47:01 dist-test-slave-jq4t krb5kdc[31187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528021, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.2@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.99.3@KRBTEST.COM; defaulting to no policy
I20260812 09:47:01.698635 32371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:47:01.674285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.2:57441 (local address 127.13.99.62:45007)
0812 09:47:01.674416 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0812 09:47:01.674421 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:47:01.682036 (+ 7615us) server_negotiation.cc:407] Connection header received
0812 09:47:01.683333 (+ 1297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:47:01.683337 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:47:01.683406 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:47:01.683463 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:47:01.685227 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:47:01.686106 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:47:01.690087 (+ 3981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:47:01.690470 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:47:01.690549 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:47:01.693776 (+ 3227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:47:01.693782 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:47:01.693788 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:47:01.693827 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:47:01.695767 (+ 1940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:47:01.696463 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:47:01.696468 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:47:01.696472 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:47:01.696595 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:47:01.697056 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:47:01.697061 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:47:01.697066 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:47:01.697348 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:47:01.697450 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0812 09:47:01.698302 (+ 852us) server_negotiation.cc:299] Negotiation successful
0812 09:47:01.698457 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260812 09:47:01.700115 32611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:47:01.674729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:45007 (local address 127.13.99.2:57441)
0812 09:47:01.681875 (+ 7146us) negotiation.cc:107] Waiting for socket to connect
0812 09:47:01.681936 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0812 09:47:01.683083 (+ 1147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:47:01.683900 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:47:01.683909 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:47:01.684284 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:47:01.685041 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:47:01.685063 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:47:01.686275 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:47:01.686279 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:47:01.687690 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:47:01.687702 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:47:01.690765 (+ 3063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:47:01.690769 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:47:01.691000 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:47:01.691743 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:47:01.691765 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:47:01.693552 (+ 1787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:47:01.695919 (+ 2367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:47:01.695928 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:47:01.695947 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:47:01.696329 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:47:01.696722 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:47:01.696726 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:47:01.696729 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:47:01.696926 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:47:01.697535 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:47:01.697546 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:47:01.697974 (+ 428us) client_negotiation.cc:769] Sending connection context
0812 09:47:01.698550 (+ 576us) client_negotiation.cc:240] Negotiation successful
0812 09:47:01.699185 (+ 635us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6904,"thread_start_us":2040,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.99.3@KRBTEST.COM" created.
I20260812 09:47:01.701803 32608 heartbeater.cc:347] Connected to a master server at 127.13.99.62:45007
I20260812 09:47:01.702060 32608 heartbeater.cc:464] Registering TS with master...
I20260812 09:47:01.702665 32608 heartbeater.cc:511] Master 127.13.99.62:45007 requested a full tablet report, sending...
I20260812 09:47:01.703833 31977 ts_manager.cc:194] Registered new tserver with Master: bbbc574ad5fa4517b51c85f92b9fe117 (127.13.99.2:45283)
I20260812 09:47:01.704424 31977 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.99.2@KRBTEST.COM'} at 127.13.99.2:57441
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.99.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.3.keytab.
Entry for principal kudu/127.13.99.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.3.keytab.
I20260812 09:47:01.721648 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.99.20:51121/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.99.3:0
--local_ip_for_outbound_sockets=127.13.99.3
--webserver_interface=127.13.99.3
--webserver_port=0
--tserver_master_addrs=127.13.99.62:45007
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.3.keytab
--principal=kudu/127.13.99.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:39543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:47:01.907998 32615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:47:01.908360 32615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:47:01.908445 32615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:47:01.915050 32615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:47:01.915125 32615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:47:01.915215 32615 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.99.3
I20260812 09:47:01.923072 32615 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:39543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.13.99.20:51121/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.3.keytab
--principal=kudu/127.13.99.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.99.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.99.3
--webserver_port=0
--tserver_master_addrs=127.13.99.62:45007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32615
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.99.3
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:47:01.924456 32615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:47:01.925877 32615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:47:01.939759 32626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:47:01.939774 32625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:47:01.940027 32629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:47:01.941293 32615 server_base.cc:1034] running on GCE node
I20260812 09:47:01.941886 32615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:47:01.942744 32615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:47:01.943954 32615 hybrid_clock.cc:648] HybridClock initialized: now 1786528021943884 us; error 85 us; skew 500 ppm
Aug 12 09:47:01 dist-test-slave-jq4t krb5kdc[31187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528021, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:47:01.947717 32615 init.cc:377] Logged in from keytab as kudu/127.13.99.3@KRBTEST.COM (short username kudu)
I20260812 09:47:01.948875 32615 webserver.cc:460] Webserver started at http://127.13.99.3:43317/ using document root <none> and password file <none>
I20260812 09:47:01.949613 32615 fs_manager.cc:359] Metadata directory not provided
I20260812 09:47:01.949705 32615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:47:01.949976 32615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 09:47:01 dist-test-slave-jq4t krb5kdc[31187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528021, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.3@KRBTEST.COM for HTTP/127.13.99.20@KRBTEST.COM
I20260812 09:47:01.975914 32615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/minicluster-data/ts-2/data/instance:
uuid: "20ebd902150547c2b095fc470daca9f4"
format_stamp: "Formatted at 2026-08-12 09:47:01 on dist-test-slave-jq4t"
server_key: "98d59fdbc89e84ce93820d75d8027b1c"
server_key_iv: "4eb944abae1ac32a1064c0e6ad7b18db"
server_key_version: "kuduclusterkey@0"
I20260812 09:47:01.976739 32615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/minicluster-data/ts-2/wal/instance:
uuid: "20ebd902150547c2b095fc470daca9f4"
format_stamp: "Formatted at 2026-08-12 09:47:01 on dist-test-slave-jq4t"
server_key: "98d59fdbc89e84ce93820d75d8027b1c"
server_key_iv: "4eb944abae1ac32a1064c0e6ad7b18db"
server_key_version: "kuduclusterkey@0"
I20260812 09:47:01.982707 32615 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260812 09:47:02.012825 32639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:47:02.014557 32615 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260812 09:47:02.014722 32615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/minicluster-data/ts-2/wal
uuid: "20ebd902150547c2b095fc470daca9f4"
format_stamp: "Formatted at 2026-08-12 09:47:01 on dist-test-slave-jq4t"
server_key: "98d59fdbc89e84ce93820d75d8027b1c"
server_key_iv: "4eb944abae1ac32a1064c0e6ad7b18db"
server_key_version: "kuduclusterkey@0"
I20260812 09:47:02.014879 32615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Aug 12 09:47:02 dist-test-slave-jq4t krb5kdc[31187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528021, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.62@KRBTEST.COM for HTTP/127.13.99.20@KRBTEST.COM
I20260812 09:47:02.083374 32615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:47:02.089697 32615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:47:02.090013 32615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:47:02.090842 32615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 09:47:02.094270 32615 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:47:02.096511 32615 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260812 09:47:02.096652 32615 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:47:02.096736 32615 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 09:47:02.118830 32615 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.3:42865
I20260812 09:47:02.120554 32615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/minicluster-data/ts-2/data/info.pb
I20260812 09:47:02.129940 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 32615
I20260812 09:47:02.130080 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/minicluster-data/ts-2/wal/instance
I20260812 09:47:02.130420 13708 external_mini_cluster.cc:1428] Setting key b2ffb5f1e2b4aee4b9a8275ff2285136
I20260812 09:47:02.131301 378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.3:42865 every 8 connection(s)
Aug 12 09:47:02 dist-test-slave-jq4t krb5kdc[31187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528021, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.3@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
I20260812 09:47:02.163625 32371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:47:02.145783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.3:60727 (local address 127.13.99.62:45007)
0812 09:47:02.145907 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0812 09:47:02.145912 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:47:02.147829 (+ 1917us) server_negotiation.cc:407] Connection header received
0812 09:47:02.149040 (+ 1211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:47:02.149043 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:47:02.149097 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:47:02.149185 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:47:02.150872 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:47:02.151751 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:47:02.153661 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:47:02.154010 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:47:02.154098 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:47:02.158610 (+ 4512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:47:02.158614 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:47:02.158617 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:47:02.158647 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:47:02.160631 (+ 1984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:47:02.161374 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:47:02.161378 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:47:02.161380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:47:02.161489 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:47:02.162065 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:47:02.162069 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:47:02.162071 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:47:02.162312 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:47:02.162413 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0812 09:47:02.163287 (+ 874us) server_negotiation.cc:299] Negotiation successful
0812 09:47:02.163442 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260812 09:47:02.165606 388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:47:02.146443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:45007 (local address 127.13.99.3:60727)
0812 09:47:02.147672 (+ 1229us) negotiation.cc:107] Waiting for socket to connect
0812 09:47:02.147732 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0812 09:47:02.148816 (+ 1084us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:47:02.149534 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:47:02.149544 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:47:02.149919 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:47:02.150672 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:47:02.150691 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:47:02.151920 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:47:02.151923 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:47:02.153515 (+ 1592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:47:02.153527 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:47:02.154310 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:47:02.154314 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:47:02.154521 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:47:02.155824 (+ 1303us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:47:02.155845 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:47:02.158383 (+ 2538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:47:02.160857 (+ 2474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:47:02.160864 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:47:02.160881 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:47:02.161246 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:47:02.161605 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:47:02.161608 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:47:02.161610 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:47:02.161948 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:47:02.162491 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:47:02.162498 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:47:02.162846 (+ 348us) client_negotiation.cc:769] Sending connection context
0812 09:47:02.164100 (+ 1254us) client_negotiation.cc:240] Negotiation successful
0812 09:47:02.164491 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1005,"thread_start_us":131,"threads_started":1}
I20260812 09:47:02.167435 379 heartbeater.cc:347] Connected to a master server at 127.13.99.62:45007
I20260812 09:47:02.167687 379 heartbeater.cc:464] Registering TS with master...
I20260812 09:47:02.168290 379 heartbeater.cc:511] Master 127.13.99.62:45007 requested a full tablet report, sending...
I20260812 09:47:02.170082 31976 ts_manager.cc:194] Registered new tserver with Master: 20ebd902150547c2b095fc470daca9f4 (127.13.99.3:42865)
I20260812 09:47:02.170886 31976 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.99.3@KRBTEST.COM'} at 127.13.99.3:60727
I20260812 09:47:02.171650 13708 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 09:47:02.183557 32371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:47:02.175615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38012 (local address 127.13.99.62:45007)
0812 09:47:02.175746 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0812 09:47:02.175750 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:47:02.176044 (+ 294us) server_negotiation.cc:407] Connection header received
0812 09:47:02.176202 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:47:02.176206 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:47:02.176255 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:47:02.176344 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:47:02.177187 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:47:02.177705 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:47:02.178552 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:47:02.178768 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:47:02.178874 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:47:02.180295 (+ 1421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:47:02.180299 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:47:02.180302 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:47:02.180326 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:47:02.181894 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:47:02.182314 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:47:02.182319 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:47:02.182321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:47:02.182410 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:47:02.182689 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:47:02.182694 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:47:02.182696 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:47:02.182870 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:47:02.183009 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0812 09:47:02.183244 (+ 235us) server_negotiation.cc:299] Negotiation successful
0812 09:47:02.183358 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260812 09:47:02.186583 31977 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38012:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 09:47:02.189833 31977 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 09:47:02.222997 404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:47:02.205225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.1:38159 (local address 127.0.0.1:36948)
0812 09:47:02.212948 (+ 7723us) negotiation.cc:107] Waiting for socket to connect
0812 09:47:02.212963 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 09:47:02.213071 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:47:02.214111 (+ 1040us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:47:02.214114 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:47:02.214136 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:47:02.214296 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:47:02.214303 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:47:02.216394 (+ 2091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:47:02.216398 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:47:02.220946 (+ 4548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:47:02.220956 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:47:02.222379 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:47:02.222383 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:47:02.222517 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:47:02.222563 (+ 46us) client_negotiation.cc:769] Sending connection context
0812 09:47:02.222688 (+ 125us) client_negotiation.cc:240] Negotiation successful
0812 09:47:02.222824 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7659,"mutex_wait_us":1,"thread_start_us":95,"threads_started":1}
I20260812 09:47:02.224999 407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:47:02.204421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.2:45283 (local address 127.0.0.1:35340)
0812 09:47:02.211406 (+ 6985us) negotiation.cc:107] Waiting for socket to connect
0812 09:47:02.211476 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0812 09:47:02.211649 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:47:02.214419 (+ 2770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:47:02.214421 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:47:02.214438 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:47:02.214514 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:47:02.214518 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:47:02.219616 (+ 5098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:47:02.219619 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:47:02.220760 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:47:02.220772 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:47:02.224576 (+ 3804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:47:02.224583 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:47:02.224679 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:47:02.224701 (+ 22us) client_negotiation.cc:769] Sending connection context
0812 09:47:02.224773 (+ 72us) client_negotiation.cc:240] Negotiation successful
0812 09:47:02.224857 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6758,"thread_start_us":138,"threads_started":1}
I20260812 09:47:02.226037 406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:47:02.205576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36948 (local address 127.13.99.1:38159)
0812 09:47:02.211076 (+ 5500us) server_negotiation.cc:206] Beginning negotiation
0812 09:47:02.211084 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 09:47:02.213485 (+ 2401us) server_negotiation.cc:407] Connection header received
0812 09:47:02.213578 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:47:02.213584 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:47:02.213786 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:47:02.213936 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:47:02.214649 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:47:02.216250 (+ 1601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:47:02.221091 (+ 4841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:47:02.221991 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:47:02.222122 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:47:02.222302 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0812 09:47:02.225764 (+ 3462us) server_negotiation.cc:299] Negotiation successful
0812 09:47:02.225884 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5351,"thread_start_us":106,"threads_started":1}
I20260812 09:47:02.231062 32233 tablet_service.cc:1467] Processing CreateTablet for tablet 120da7132820402b93f16b21526717f8 (DEFAULT_TABLE table=test-table [id=eafcce7f7ab64ec5bcd023b8f3c7cd11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 09:47:02.232448 32233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 120da7132820402b93f16b21526717f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:47:02.241086 405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:47:02.205585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.3:42865 (local address 127.0.0.1:38564)
0812 09:47:02.213249 (+ 7664us) negotiation.cc:107] Waiting for socket to connect
0812 09:47:02.213262 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 09:47:02.213347 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:47:02.214092 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:47:02.214098 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:47:02.214123 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:47:02.215084 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:47:02.215095 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:47:02.226259 (+ 11164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:47:02.226263 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:47:02.227450 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:47:02.227462 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:47:02.240567 (+ 13105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:47:02.240573 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:47:02.240700 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:47:02.240732 (+ 32us) client_negotiation.cc:769] Sending connection context
0812 09:47:02.240830 (+ 98us) client_negotiation.cc:240] Negotiation successful
0812 09:47:02.240920 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7608,"mutex_wait_us":686,"thread_start_us":111,"threads_started":1}
I20260812 09:47:02.241195 412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:47:02.205831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38564 (local address 127.13.99.3:42865)
0812 09:47:02.212775 (+ 6944us) server_negotiation.cc:206] Beginning negotiation
0812 09:47:02.212788 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 09:47:02.213451 (+ 663us) server_negotiation.cc:407] Connection header received
0812 09:47:02.213545 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:47:02.213551 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:47:02.213742 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:47:02.213908 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:47:02.224572 (+ 10664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:47:02.225708 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:47:02.227605 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:47:02.228403 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:47:02.228534 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:47:02.228676 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0812 09:47:02.240903 (+ 12227us) server_negotiation.cc:299] Negotiation successful
0812 09:47:02.241041 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6820,"thread_start_us":196,"threads_started":1}
I20260812 09:47:02.242069 410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:47:02.205913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35340 (local address 127.13.99.2:45283)
0812 09:47:02.206213 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0812 09:47:02.206222 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 09:47:02.212766 (+ 6544us) server_negotiation.cc:407] Connection header received
0812 09:47:02.212865 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:47:02.212871 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:47:02.213148 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:47:02.213314 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:47:02.217538 (+ 4224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:47:02.219458 (+ 1920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:47:02.223368 (+ 3910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:47:02.224219 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:47:02.224331 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:47:02.224510 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0812 09:47:02.241785 (+ 17275us) server_negotiation.cc:299] Negotiation successful
0812 09:47:02.241913 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":100,"threads_started":1}
I20260812 09:47:02.244750 32726 tablet_service.cc:1467] Processing CreateTablet for tablet 120da7132820402b93f16b21526717f8 (DEFAULT_TABLE table=test-table [id=eafcce7f7ab64ec5bcd023b8f3c7cd11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 09:47:02.246057 32726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 120da7132820402b93f16b21526717f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:47:02.256119 32498 tablet_service.cc:1467] Processing CreateTablet for tablet 120da7132820402b93f16b21526717f8 (DEFAULT_TABLE table=test-table [id=eafcce7f7ab64ec5bcd023b8f3c7cd11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 09:47:02.257354 32498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 120da7132820402b93f16b21526717f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:47:02.258347 413 tablet_bootstrap.cc:492] T 120da7132820402b93f16b21526717f8 P 20ebd902150547c2b095fc470daca9f4: Bootstrap starting.
I20260812 09:47:02.261389 413 tablet_bootstrap.cc:654] T 120da7132820402b93f16b21526717f8 P 20ebd902150547c2b095fc470daca9f4: Neither blocks nor log segments found. Creating new log.
I20260812 09:47:02.262390 413 log.cc:824] T 120da7132820402b93f16b21526717f8 P 20ebd902150547c2b095fc470daca9f4: Log is configured to *not* fsync() on all Append() calls
I20260812 09:47:02.264778 413 tablet_bootstrap.cc:492] T 120da7132820402b93f16b21526717f8 P 20ebd902150547c2b095fc470daca9f4: No bootstrap required, opened a new log
I20260812 09:47:02.265010 413 ts_tablet_manager.cc:1397] T 120da7132820402b93f16b21526717f8 P 20ebd902150547c2b095fc470daca9f4: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260812 09:47:02.266222 414 tablet_bootstrap.cc:492] T 120da7132820402b93f16b21526717f8 P efbbdd7325f740b3abc76c509ba76e4b: Bootstrap starting.
I20260812 09:47:02.268672 414 tablet_bootstrap.cc:654] T 120da7132820402b93f16b21526717f8 P efbbdd7325f740b3abc76c509ba76e4b: Neither blocks nor log segments found. Creating new log.
I20260812 09:47:02.269343 413 raft_consensus.cc:348] T 120da7132820402b93f16b21526717f8 P 20ebd902150547c2b095fc470daca9f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbbc574ad5fa4517b51c85f92b9fe117" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 45283 } } peers { permanent_uuid: "efbbdd7325f740b3abc76c509ba76e4b" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 38159 } } peers { permanent_uuid: "20ebd902150547c2b095fc470daca9f4" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 42865 } }
I20260812 09:47:02.269608 414 log.cc:824] T 120da7132820402b93f16b21526717f8 P efbbdd7325f740b3abc76c509ba76e4b: Log is configured to *not* fsync() on all Append() calls
I20260812 09:47:02.269732 413 raft_consensus.cc:374] T 120da7132820402b93f16b21526717f8 P 20ebd902150547c2b095fc470daca9f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:47:02.269802 413 raft_consensus.cc:729] T 120da7132820402b93f16b21526717f8 P 20ebd902150547c2b095fc470daca9f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20ebd902150547c2b095fc470daca9f4, State: Initialized, Role: FOLLOWER
I20260812 09:47:02.270334 413 consensus_queue.cc:260] T 120da7132820402b93f16b21526717f8 P 20ebd902150547c2b095fc470daca9f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbbc574ad5fa4517b51c85f92b9fe117" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 45283 } } peers { permanent_uuid: "efbbdd7325f740b3abc76c509ba76e4b" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 38159 } } peers { permanent_uuid: "20ebd902150547c2b095fc470daca9f4" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 42865 } }
I20260812 09:47:02.271466 413 ts_tablet_manager.cc:1428] T 120da7132820402b93f16b21526717f8 P 20ebd902150547c2b095fc470daca9f4: Time spent starting tablet: real 0.006s user 0.001s sys 0.004s
I20260812 09:47:02.271869 379 heartbeater.cc:503] Master 127.13.99.62:45007 was elected leader, sending a full tablet report...
I20260812 09:47:02.277544 417 tablet_bootstrap.cc:492] T 120da7132820402b93f16b21526717f8 P bbbc574ad5fa4517b51c85f92b9fe117: Bootstrap starting.
I20260812 09:47:02.279565 414 tablet_bootstrap.cc:492] T 120da7132820402b93f16b21526717f8 P efbbdd7325f740b3abc76c509ba76e4b: No bootstrap required, opened a new log
I20260812 09:47:02.279834 414 ts_tablet_manager.cc:1397] T 120da7132820402b93f16b21526717f8 P efbbdd7325f740b3abc76c509ba76e4b: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.003s
I20260812 09:47:02.280450 417 tablet_bootstrap.cc:654] T 120da7132820402b93f16b21526717f8 P bbbc574ad5fa4517b51c85f92b9fe117: Neither blocks nor log segments found. Creating new log.
I20260812 09:47:02.281358 417 log.cc:824] T 120da7132820402b93f16b21526717f8 P bbbc574ad5fa4517b51c85f92b9fe117: Log is configured to *not* fsync() on all Append() calls
I20260812 09:47:02.284819 414 raft_consensus.cc:348] T 120da7132820402b93f16b21526717f8 P efbbdd7325f740b3abc76c509ba76e4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbbc574ad5fa4517b51c85f92b9fe117" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 45283 } } peers { permanent_uuid: "efbbdd7325f740b3abc76c509ba76e4b" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 38159 } } peers { permanent_uuid: "20ebd902150547c2b095fc470daca9f4" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 42865 } }
I20260812 09:47:02.285168 414 raft_consensus.cc:374] T 120da7132820402b93f16b21526717f8 P efbbdd7325f740b3abc76c509ba76e4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:47:02.285280 414 raft_consensus.cc:729] T 120da7132820402b93f16b21526717f8 P efbbdd7325f740b3abc76c509ba76e4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efbbdd7325f740b3abc76c509ba76e4b, State: Initialized, Role: FOLLOWER
I20260812 09:47:02.285853 414 consensus_queue.cc:260] T 120da7132820402b93f16b21526717f8 P efbbdd7325f740b3abc76c509ba76e4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbbc574ad5fa4517b51c85f92b9fe117" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 45283 } } peers { permanent_uuid: "efbbdd7325f740b3abc76c509ba76e4b" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 38159 } } peers { permanent_uuid: "20ebd902150547c2b095fc470daca9f4" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 42865 } }
I20260812 09:47:02.287067 414 ts_tablet_manager.cc:1428] T 120da7132820402b93f16b21526717f8 P efbbdd7325f740b3abc76c509ba76e4b: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20260812 09:47:02.287472 32364 heartbeater.cc:503] Master 127.13.99.62:45007 was elected leader, sending a full tablet report...
I20260812 09:47:02.292773 417 tablet_bootstrap.cc:492] T 120da7132820402b93f16b21526717f8 P bbbc574ad5fa4517b51c85f92b9fe117: No bootstrap required, opened a new log
I20260812 09:47:02.292979 417 ts_tablet_manager.cc:1397] T 120da7132820402b93f16b21526717f8 P bbbc574ad5fa4517b51c85f92b9fe117: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.002s
I20260812 09:47:02.298625 417 raft_consensus.cc:348] T 120da7132820402b93f16b21526717f8 P bbbc574ad5fa4517b51c85f92b9fe117 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbbc574ad5fa4517b51c85f92b9fe117" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 45283 } } peers { permanent_uuid: "efbbdd7325f740b3abc76c509ba76e4b" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 38159 } } peers { permanent_uuid: "20ebd902150547c2b095fc470daca9f4" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 42865 } }
I20260812 09:47:02.298977 417 raft_consensus.cc:374] T 120da7132820402b93f16b21526717f8 P bbbc574ad5fa4517b51c85f92b9fe117 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:47:02.299132 417 raft_consensus.cc:729] T 120da7132820402b93f16b21526717f8 P bbbc574ad5fa4517b51c85f92b9fe117 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbbc574ad5fa4517b51c85f92b9fe117, State: Initialized, Role: FOLLOWER
I20260812 09:47:02.299727 417 consensus_queue.cc:260] T 120da7132820402b93f16b21526717f8 P bbbc574ad5fa4517b51c85f92b9fe117 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbbc574ad5fa4517b51c85f92b9fe117" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 45283 } } peers { permanent_uuid: "efbbdd7325f740b3abc76c509ba76e4b" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 38159 } } peers { permanent_uuid: "20ebd902150547c2b095fc470daca9f4" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 42865 } }
I20260812 09:47:02.300956 417 ts_tablet_manager.cc:1428] T 120da7132820402b93f16b21526717f8 P bbbc574ad5fa4517b51c85f92b9fe117: Time spent starting tablet: real 0.008s user 0.004s sys 0.002s
I20260812 09:47:02.301371 32608 heartbeater.cc:503] Master 127.13.99.62:45007 was elected leader, sending a full tablet report...
I20260812 09:47:02.317586 418 raft_consensus.cc:482] T 120da7132820402b93f16b21526717f8 P 20ebd902150547c2b095fc470daca9f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 09:47:02.317831 418 raft_consensus.cc:504] T 120da7132820402b93f16b21526717f8 P 20ebd902150547c2b095fc470daca9f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbbc574ad5fa4517b51c85f92b9fe117" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 45283 } } peers { permanent_uuid: "efbbdd7325f740b3abc76c509ba76e4b" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 38159 } } peers { permanent_uuid: "20ebd902150547c2b095fc470daca9f4" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 42865 } }
I20260812 09:47:02.319243 418 leader_election.cc:288] T 120da7132820402b93f16b21526717f8 P 20ebd902150547c2b095fc470daca9f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbbc574ad5fa4517b51c85f92b9fe117 (127.13.99.2:45283), efbbdd7325f740b3abc76c509ba76e4b (127.13.99.1:38159)
I20260812 09:47:02.325416 388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:47:02.319821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.2:45283 (local address 127.13.99.3:42853)
0812 09:47:02.319991 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0812 09:47:02.320010 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 09:47:02.320121 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:47:02.321396 (+ 1275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:47:02.321401 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:47:02.321424 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:47:02.321556 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:47:02.321564 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:47:02.322644 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:47:02.322647 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:47:02.323913 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:47:02.323925 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:47:02.324999 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:47:02.325003 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:47:02.325110 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:47:02.325135 (+ 25us) client_negotiation.cc:769] Sending connection context
0812 09:47:02.325210 (+ 75us) client_negotiation.cc:240] Negotiation successful
0812 09:47:02.325281 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260812 09:47:02.326246 410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:47:02.320630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.3:42853 (local address 127.13.99.2:45283)
0812 09:47:02.320921 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0812 09:47:02.320925 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:47:02.320940 (+ 15us) server_negotiation.cc:407] Connection header received
0812 09:47:02.320993 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:47:02.320995 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:47:02.321046 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:47:02.321127 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:47:02.321694 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:47:02.322492 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:47:02.324104 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:47:02.324766 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:47:02.324872 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:47:02.324938 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0812 09:47:02.326039 (+ 1101us) server_negotiation.cc:299] Negotiation successful
0812 09:47:02.326123 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211}
I20260812 09:47:02.326962 32525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "120da7132820402b93f16b21526717f8" candidate_uuid: "20ebd902150547c2b095fc470daca9f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbbc574ad5fa4517b51c85f92b9fe117" is_pre_election: true
I20260812 09:47:02.327301 32525 raft_consensus.cc:2454] T 120da7132820402b93f16b21526717f8 P bbbc574ad5fa4517b51c85f92b9fe117 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20ebd902150547c2b095fc470daca9f4 in term 0.
I20260812 09:47:02.327807 421 raft_consensus.cc:482] T 120da7132820402b93f16b21526717f8 P efbbdd7325f740b3abc76c509ba76e4b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 09:47:02.327963 421 raft_consensus.cc:504] T 120da7132820402b93f16b21526717f8 P efbbdd7325f740b3abc76c509ba76e4b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbbc574ad5fa4517b51c85f92b9fe117" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 45283 } } peers { permanent_uuid: "efbbdd7325f740b3abc76c509ba76e4b" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 38159 } } peers { permanent_uuid: "20ebd902150547c2b095fc470daca9f4" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 42865 } }
I20260812 09:47:02.328544 406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:47:02.320788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.3:45149 (local address 127.13.99.1:38159)
0812 09:47:02.321303 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0812 09:47:02.321307 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:47:02.324465 (+ 3158us) server_negotiation.cc:407] Connection header received
0812 09:47:02.324682 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:47:02.324685 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:47:02.324735 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:47:02.324781 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:47:02.325176 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:47:02.325904 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:47:02.327297 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:47:02.327865 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:47:02.327939 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:47:02.327998 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0812 09:47:02.328335 (+ 337us) server_negotiation.cc:299] Negotiation successful
0812 09:47:02.328417 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424}
I20260812 09:47:02.328881 426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:47:02.320068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.1:38159 (local address 127.13.99.3:45149)
0812 09:47:02.324354 (+ 4286us) negotiation.cc:107] Waiting for socket to connect
0812 09:47:02.324378 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 09:47:02.324561 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:47:02.324910 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:47:02.324913 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:47:02.324929 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:47:02.325055 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:47:02.325061 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:47:02.326049 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:47:02.326052 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:47:02.327155 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:47:02.327166 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:47:02.328116 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:47:02.328120 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:47:02.328210 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:47:02.328231 (+ 21us) client_negotiation.cc:769] Sending connection context
0812 09:47:02.328665 (+ 434us) client_negotiation.cc:240] Negotiation successful
0812 09:47:02.328735 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4212,"thread_start_us":124,"threads_started":1}
I20260812 09:47:02.329852 421 leader_election.cc:288] T 120da7132820402b93f16b21526717f8 P efbbdd7325f740b3abc76c509ba76e4b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbbc574ad5fa4517b51c85f92b9fe117 (127.13.99.2:45283), 20ebd902150547c2b095fc470daca9f4 (127.13.99.3:42865)
I20260812 09:47:02.329908 32651 leader_election.cc:302] T 120da7132820402b93f16b21526717f8 P 20ebd902150547c2b095fc470daca9f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20ebd902150547c2b095fc470daca9f4, bbbc574ad5fa4517b51c85f92b9fe117; no voters:
I20260812 09:47:02.330202 418 raft_consensus.cc:2781] T 120da7132820402b93f16b21526717f8 P 20ebd902150547c2b095fc470daca9f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 09:47:02.330179 32279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "120da7132820402b93f16b21526717f8" candidate_uuid: "20ebd902150547c2b095fc470daca9f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efbbdd7325f740b3abc76c509ba76e4b" is_pre_election: true
I20260812 09:47:02.330269 418 raft_consensus.cc:482] T 120da7132820402b93f16b21526717f8 P 20ebd902150547c2b095fc470daca9f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 09:47:02.330305 418 raft_consensus.cc:3037] T 120da7132820402b93f16b21526717f8 P 20ebd902150547c2b095fc470daca9f4 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:47:02.330377 32279 raft_consensus.cc:2454] T 120da7132820402b93f16b21526717f8 P efbbdd7325f740b3abc76c509ba76e4b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20ebd902150547c2b095fc470daca9f4 in term 0.
I20260812 09:47:02.331825 418 raft_consensus.cc:504] T 120da7132820402b93f16b21526717f8 P 20ebd902150547c2b095fc470daca9f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbbc574ad5fa4517b51c85f92b9fe117" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 45283 } } peers { permanent_uuid: "efbbdd7325f740b3abc76c509ba76e4b" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 38159 } } peers { permanent_uuid: "20ebd902150547c2b095fc470daca9f4" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 42865 } }
I20260812 09:47:02.332131 418 leader_election.cc:288] T 120da7132820402b93f16b21526717f8 P 20ebd902150547c2b095fc470daca9f4 [CANDIDATE]: Term 1 election: Requested vote from peers bbbc574ad5fa4517b51c85f92b9fe117 (127.13.99.2:45283), efbbdd7325f740b3abc76c509ba76e4b (127.13.99.1:38159)
I20260812 09:47:02.332683 32525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "120da7132820402b93f16b21526717f8" candidate_uuid: "20ebd902150547c2b095fc470daca9f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbbc574ad5fa4517b51c85f92b9fe117"
I20260812 09:47:02.332813 32525 raft_consensus.cc:3037] T 120da7132820402b93f16b21526717f8 P bbbc574ad5fa4517b51c85f92b9fe117 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:47:02.332839 32279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "120da7132820402b93f16b21526717f8" candidate_uuid: "20ebd902150547c2b095fc470daca9f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efbbdd7325f740b3abc76c509ba76e4b"
I20260812 09:47:02.332962 32279 raft_consensus.cc:3037] T 120da7132820402b93f16b21526717f8 P efbbdd7325f740b3abc76c509ba76e4b [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:47:02.334151 32525 raft_consensus.cc:2454] T 120da7132820402b93f16b21526717f8 P bbbc574ad5fa4517b51c85f92b9fe117 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20ebd902150547c2b095fc470daca9f4 in term 1.
I20260812 09:47:02.334307 32279 raft_consensus.cc:2454] T 120da7132820402b93f16b21526717f8 P efbbdd7325f740b3abc76c509ba76e4b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20ebd902150547c2b095fc470daca9f4 in term 1.
I20260812 09:47:02.334530 32651 leader_election.cc:302] T 120da7132820402b93f16b21526717f8 P 20ebd902150547c2b095fc470daca9f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20ebd902150547c2b095fc470daca9f4, bbbc574ad5fa4517b51c85f92b9fe117; no voters:
I20260812 09:47:02.334821 418 raft_consensus.cc:2781] T 120da7132820402b93f16b21526717f8 P 20ebd902150547c2b095fc470daca9f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 09:47:02.335050 418 raft_consensus.cc:686] T 120da7132820402b93f16b21526717f8 P 20ebd902150547c2b095fc470daca9f4 [term 1 LEADER]: Becoming Leader. State: Replica: 20ebd902150547c2b095fc470daca9f4, State: Running, Role: LEADER
I20260812 09:47:02.335430 418 consensus_queue.cc:237] T 120da7132820402b93f16b21526717f8 P 20ebd902150547c2b095fc470daca9f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbbc574ad5fa4517b51c85f92b9fe117" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 45283 } } peers { permanent_uuid: "efbbdd7325f740b3abc76c509ba76e4b" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 38159 } } peers { permanent_uuid: "20ebd902150547c2b095fc470daca9f4" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 42865 } }
I20260812 09:47:02.338773 31977 catalog_manager.cc:5495] T 120da7132820402b93f16b21526717f8 P 20ebd902150547c2b095fc470daca9f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 20ebd902150547c2b095fc470daca9f4 (127.13.99.3). New cstate: current_term: 1 leader_uuid: "20ebd902150547c2b095fc470daca9f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbbc574ad5fa4517b51c85f92b9fe117" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 45283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efbbdd7325f740b3abc76c509ba76e4b" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 38159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20ebd902150547c2b095fc470daca9f4" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 42865 } health_report { overall_health: HEALTHY } } }
I20260812 09:47:02.345679 410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:47:02.331710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.1:40857 (local address 127.13.99.2:45283)
0812 09:47:02.331866 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0812 09:47:02.331870 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:47:02.334941 (+ 3071us) server_negotiation.cc:407] Connection header received
0812 09:47:02.335181 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:47:02.335185 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:47:02.335233 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:47:02.335286 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:47:02.335785 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:47:02.336547 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:47:02.344281 (+ 7734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:47:02.344918 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:47:02.345002 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:47:02.345087 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0812 09:47:02.345463 (+ 376us) server_negotiation.cc:299] Negotiation successful
0812 09:47:02.345549 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260812 09:47:02.345811 412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:47:02.331661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.1:42097 (local address 127.13.99.3:42865)
0812 09:47:02.331765 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0812 09:47:02.331771 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 09:47:02.340341 (+ 8570us) server_negotiation.cc:407] Connection header received
0812 09:47:02.341505 (+ 1164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:47:02.341508 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:47:02.341554 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:47:02.341602 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:47:02.342039 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:47:02.342731 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:47:02.344269 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:47:02.345064 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:47:02.345183 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:47:02.345244 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0812 09:47:02.345593 (+ 349us) server_negotiation.cc:299] Negotiation successful
0812 09:47:02.345681 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260812 09:47:02.346179 427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:47:02.331293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.3:42865 (local address 127.13.99.1:42097)
0812 09:47:02.340243 (+ 8950us) negotiation.cc:107] Waiting for socket to connect
0812 09:47:02.340261 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 09:47:02.341388 (+ 1127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:47:02.341751 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:47:02.341754 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:47:02.341771 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:47:02.341888 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:47:02.341894 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:47:02.342879 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:47:02.342882 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:47:02.344112 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:47:02.344124 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:47:02.345317 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:47:02.345322 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:47:02.345434 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:47:02.345459 (+ 25us) client_negotiation.cc:769] Sending connection context
0812 09:47:02.345952 (+ 493us) client_negotiation.cc:240] Negotiation successful
0812 09:47:02.346031 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8877,"thread_start_us":88,"threads_started":1}
I20260812 09:47:02.346400 433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:47:02.330846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.2:45283 (local address 127.13.99.1:40857)
0812 09:47:02.334843 (+ 3997us) negotiation.cc:107] Waiting for socket to connect
0812 09:47:02.334858 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 09:47:02.335012 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:47:02.335429 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:47:02.335432 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:47:02.335451 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:47:02.335645 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:47:02.335655 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:47:02.336735 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:47:02.336739 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:47:02.341238 (+ 4499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:47:02.341251 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:47:02.345196 (+ 3945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:47:02.345200 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:47:02.345302 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:47:02.345326 (+ 24us) client_negotiation.cc:769] Sending connection context
0812 09:47:02.346180 (+ 854us) client_negotiation.cc:240] Negotiation successful
0812 09:47:02.346262 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3899,"thread_start_us":169,"threads_started":1}
I20260812 09:47:02.346948 32759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "120da7132820402b93f16b21526717f8" candidate_uuid: "efbbdd7325f740b3abc76c509ba76e4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20ebd902150547c2b095fc470daca9f4" is_pre_election: true
I20260812 09:47:02.347050 32525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "120da7132820402b93f16b21526717f8" candidate_uuid: "efbbdd7325f740b3abc76c509ba76e4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbbc574ad5fa4517b51c85f92b9fe117" is_pre_election: true
I20260812 09:47:02.347234 32525 raft_consensus.cc:2379] T 120da7132820402b93f16b21526717f8 P bbbc574ad5fa4517b51c85f92b9fe117 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate efbbdd7325f740b3abc76c509ba76e4b in current term 1: Already voted for candidate 20ebd902150547c2b095fc470daca9f4 in this term.
I20260812 09:47:02.347843 32156 leader_election.cc:302] T 120da7132820402b93f16b21526717f8 P efbbdd7325f740b3abc76c509ba76e4b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: efbbdd7325f740b3abc76c509ba76e4b; no voters: 20ebd902150547c2b095fc470daca9f4, bbbc574ad5fa4517b51c85f92b9fe117
I20260812 09:47:02.348341 421 raft_consensus.cc:2726] T 120da7132820402b93f16b21526717f8 P efbbdd7325f740b3abc76c509ba76e4b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260812 09:47:02.361019 32368 tablet.cc:2369] T 120da7132820402b93f16b21526717f8 P efbbdd7325f740b3abc76c509ba76e4b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 09:47:02.374145 386 tablet.cc:2369] T 120da7132820402b93f16b21526717f8 P 20ebd902150547c2b095fc470daca9f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 09:47:02.374497 412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:47:02.369219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38570 (local address 127.13.99.3:42865)
0812 09:47:02.369342 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0812 09:47:02.369347 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:47:02.369398 (+ 51us) server_negotiation.cc:407] Connection header received
0812 09:47:02.369598 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:47:02.369602 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:47:02.369672 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:47:02.369791 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 09:47:02.370125 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:47:02.370901 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:47:02.372158 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:47:02.372556 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:47:02.372663 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:47:02.373125 (+ 462us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 09:47:02.373714 (+ 589us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 09:47:02.373846 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0812 09:47:02.374293 (+ 447us) server_negotiation.cc:299] Negotiation successful
0812 09:47:02.374360 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260812 09:47:02.381421 32279 raft_consensus.cc:1261] T 120da7132820402b93f16b21526717f8 P efbbdd7325f740b3abc76c509ba76e4b [term 1 FOLLOWER]: Refusing update from remote peer 20ebd902150547c2b095fc470daca9f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 09:47:02.388715 418 consensus_queue.cc:1035] T 120da7132820402b93f16b21526717f8 P 20ebd902150547c2b095fc470daca9f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efbbdd7325f740b3abc76c509ba76e4b" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 38159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 09:47:02.390231 32525 raft_consensus.cc:1261] T 120da7132820402b93f16b21526717f8 P bbbc574ad5fa4517b51c85f92b9fe117 [term 1 FOLLOWER]: Refusing update from remote peer 20ebd902150547c2b095fc470daca9f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 09:47:02.392402 418 consensus_queue.cc:1035] T 120da7132820402b93f16b21526717f8 P 20ebd902150547c2b095fc470daca9f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbbc574ad5fa4517b51c85f92b9fe117" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 45283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 09:47:02.402809 437 mvcc.cc:205] Tried to move back new op lower bound from 7317618779666210816 to 7317618779488776192. Current Snapshot: MvccSnapshot[applied={T|T < 7317618779666210816}]
Aug 12 09:47:02 dist-test-slave-jq4t krb5kdc[31187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528021, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.62@KRBTEST.COM for HTTP/127.13.99.20@KRBTEST.COM
I20260812 09:47:02.420951 31976 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38012:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 09:47:02.421135 31976 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38012:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 09:47:02.424031 31976 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2e4eae9476df441e94bd11701ebb9cf8: Sending DeleteTablet for 3 replicas of tablet 120da7132820402b93f16b21526717f8
I20260812 09:47:02.425312 32498 tablet_service.cc:1514] Processing DeleteTablet for tablet 120da7132820402b93f16b21526717f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 09:47:02 UTC) from {username='kudu', principal='kudu/127.13.99.62@KRBTEST.COM'} at 127.0.0.1:35340
I20260812 09:47:02.425858 32233 tablet_service.cc:1514] Processing DeleteTablet for tablet 120da7132820402b93f16b21526717f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 09:47:02 UTC) from {username='kudu', principal='kudu/127.13.99.62@KRBTEST.COM'} at 127.0.0.1:36948
I20260812 09:47:02.427268 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 32118
I20260812 09:47:02.428177 32726 tablet_service.cc:1514] Processing DeleteTablet for tablet 120da7132820402b93f16b21526717f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 09:47:02 UTC) from {username='kudu', principal='kudu/127.13.99.62@KRBTEST.COM'} at 127.0.0.1:38564
I20260812 09:47:02.430739 456 tablet_replica.cc:330] T 120da7132820402b93f16b21526717f8 P 20ebd902150547c2b095fc470daca9f4: stopping tablet replica
I20260812 09:47:02.431120 456 raft_consensus.cc:2229] T 120da7132820402b93f16b21526717f8 P 20ebd902150547c2b095fc470daca9f4 [term 1 LEADER]: Raft consensus shutting down.
I20260812 09:47:02.431602 456 raft_consensus.cc:2258] T 120da7132820402b93f16b21526717f8 P 20ebd902150547c2b095fc470daca9f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 09:47:02.444763 454 tablet_replica.cc:330] T 120da7132820402b93f16b21526717f8 P bbbc574ad5fa4517b51c85f92b9fe117: stopping tablet replica
I20260812 09:47:02.444948 456 ts_tablet_manager.cc:1905] T 120da7132820402b93f16b21526717f8 P 20ebd902150547c2b095fc470daca9f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 09:47:02.449079 454 raft_consensus.cc:2229] T 120da7132820402b93f16b21526717f8 P bbbc574ad5fa4517b51c85f92b9fe117 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 09:47:02.449577 454 raft_consensus.cc:2258] T 120da7132820402b93f16b21526717f8 P bbbc574ad5fa4517b51c85f92b9fe117 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 09:47:02.450848 456 ts_tablet_manager.cc:1918] T 120da7132820402b93f16b21526717f8 P 20ebd902150547c2b095fc470daca9f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 09:47:02.451329 456 log.cc:1196] T 120da7132820402b93f16b21526717f8 P 20ebd902150547c2b095fc470daca9f4: Deleting WAL directory at /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/minicluster-data/ts-2/wal/wals/120da7132820402b93f16b21526717f8
I20260812 09:47:02.452030 456 ts_tablet_manager.cc:1939] T 120da7132820402b93f16b21526717f8 P 20ebd902150547c2b095fc470daca9f4: Deleting consensus metadata
I20260812 09:47:02.453819 31956 catalog_manager.cc:4841] TS 20ebd902150547c2b095fc470daca9f4 (127.13.99.3:42865): tablet 120da7132820402b93f16b21526717f8 (table test-table [id=eafcce7f7ab64ec5bcd023b8f3c7cd11]) successfully deleted
I20260812 09:47:02.453665 454 ts_tablet_manager.cc:1905] T 120da7132820402b93f16b21526717f8 P bbbc574ad5fa4517b51c85f92b9fe117: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 09:47:02.465217 454 ts_tablet_manager.cc:1918] T 120da7132820402b93f16b21526717f8 P bbbc574ad5fa4517b51c85f92b9fe117: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 09:47:02.465458 454 log.cc:1196] T 120da7132820402b93f16b21526717f8 P bbbc574ad5fa4517b51c85f92b9fe117: Deleting WAL directory at /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786527905902985-13708-0/minicluster-data/ts-1/wal/wals/120da7132820402b93f16b21526717f8
I20260812 09:47:02.465821 454 ts_tablet_manager.cc:1939] T 120da7132820402b93f16b21526717f8 P bbbc574ad5fa4517b51c85f92b9fe117: Deleting consensus metadata
I20260812 09:47:02.467326 31956 catalog_manager.cc:4841] TS bbbc574ad5fa4517b51c85f92b9fe117 (127.13.99.2:45283): tablet 120da7132820402b93f16b21526717f8 (table test-table [id=eafcce7f7ab64ec5bcd023b8f3c7cd11]) successfully deleted
W20260812 09:47:02.468313 31956 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.99.1:38159 (error 108)
I20260812 09:47:02.469909 405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:47:02.469593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.1:38159 (local address 127.0.0.1:36952)
0812 09:47:02.469740 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0812 09:47:02.469817 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.1:38159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260812 09:47:02.470212 31956 catalog_manager.cc:4584] TS efbbdd7325f740b3abc76c509ba76e4b (127.13.99.1:38159): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 120da7132820402b93f16b21526717f8: Network error: Client connection negotiation failed: client connection to 127.13.99.1:38159: connect: Connection refused (error 111)
I20260812 09:47:02.472672 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 32377
I20260812 09:47:02.523856 405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:47:02.523339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.1:38159 (local address 127.0.0.1:36960)
0812 09:47:02.523623 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0812 09:47:02.523716 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.1:38159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157}
I20260812 09:47:02.524910 405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:47:02.524622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.1:38159 (local address 127.0.0.1:36976)
0812 09:47:02.524783 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0812 09:47:02.524835 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.1:38159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260812 09:47:02.564821 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 32615
I20260812 09:47:02.583794 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 31922
I20260812 09:47:02.616881 13708 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260812 09:47:02.961942 13708 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260812 09:47:02.962147 13708 mini_ranger.cc:66] Stopping Ranger...
I20260812 09:47:03.317135 13708 mini_ranger.cc:68] Stopped Ranger
2026-08-12 09:47:03.317 UTC [31231] LOG: received smart shutdown request
2026-08-12 09:47:03.321 UTC [31231] LOG: background worker "logical replication launcher" (PID 31244) exited with exit code 1
2026-08-12 09:47:03.322 UTC [31235] LOG: shutting down
2026-08-12 09:47:03.487 UTC [31231] LOG: database system is shut down
2026-08-12T09:47:03Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (47769 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 09:47:03 dist-test-slave-jq4t krb5kdc[475](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 09:47:03 dist-test-slave-jq4t krb5kdc[475](info): set up 2 sockets
Aug 12 09:47:03 dist-test-slave-jq4t krb5kdc[475](info): commencing operation
krb5kdc: starting...
W20260812 09:47:04.872754 13708 mini_kdc.cc:121] Time spent starting KDC: real 1.242s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/krb5kdc/test-user.keytab.
Aug 12 09:47:04 dist-test-slave-jq4t krb5kdc[475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528024, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T09:47:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T09:47:04Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.13.99.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.13.99.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.13.99.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/krb5kdc/rangeradmin_127.13.99.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.13.99.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/krb5kdc/rangeradmin_127.13.99.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.13.99.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.13.99.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.13.99.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/krb5kdc/rangerlookup_127.13.99.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.13.99.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/krb5kdc/rangerlookup_127.13.99.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.13.99.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.99.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.99.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/krb5kdc/HTTP_127.13.99.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.13.99.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/krb5kdc/HTTP_127.13.99.20@KRBTEST.COM.keytab.
I20260812 09:47:05.049152 13708 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskJsh0qK/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/postgres -l logfile start
2026-08-12 09:47:07.628 UTC [530] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-12 09:47:07.628 UTC [530] LOG: listening on IPv4 address "127.13.99.20", port 51271
2026-08-12 09:47:07.632 UTC [530] LOG: listening on Unix socket "/tmp/.s.PGSQL.51271"
2026-08-12 09:47:07.671 UTC [534] LOG: database system was shut down at 2026-08-12 09:47:06 UTC
2026-08-12 09:47:07.676 UTC [530] LOG: database system is ready to accept connections
I20260812 09:47:08.881089 13708 mini_postgres.cc:96] Postgres bound to 51271
2026-08-12 09:47:08.888 UTC [551] FATAL: database "slave" does not exist
127.13.99.20:51271 - accepting connections
I20260812 09:47:08.889362 13708 mini_ranger.cc:161] Starting Ranger...
I20260812 09:47:08.905316 13708 mini_ranger.cc:84] Created miniranger Postgres user
I20260812 09:47:09.417341 13708 mini_ranger.cc:87] Created ranger Postgres database
I20260812 09:47:09.417446 13708 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ranger-admin
2026-08-12 09:47:09,694 [I] DB FLAVOR :POSTGRES
2026-08-12 09:47:09,694 [I] --------- Verifying Ranger DB connection ---------
2026-08-12 09:47:09,694 [I] Checking connection..
2026-08-12 09:47:09,695 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJsh0qK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJsh0qK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.99.20:51271/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-12 09:47:09,944 [I] Checking connection passed.
2026-08-12 09:47:09,945 [I] --------- Verifying version history table ---------
2026-08-12 09:47:09,945 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJsh0qK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJsh0qK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.99.20:51271/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 09:47:10,220 [I] Table x_db_version_h does not exist in database ranger
2026-08-12 09:47:10,220 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJsh0qK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJsh0qK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.99.20:51271/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 09:47:10,492 [I] Table x_db_version_h does not exist in database ranger
2026-08-12 09:47:10,492 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-12 09:47:10,493 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJsh0qK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJsh0qK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.99.20:51271/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskJsh0qK/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-12 09:47:10.755 UTC [663] WARNING: there is no transaction in progress
2026-08-12 09:47:10,765 [I] create_dbversion_catalog.sql file imported successfully
2026-08-12 09:47:10,765 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJsh0qK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJsh0qK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.99.20:51271/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 09:47:11,016 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-12 09:47:11,016 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-12 09:47:11,017 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJsh0qK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJsh0qK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.99.20:51271/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-12 09:47:11,263 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJsh0qK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJsh0qK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.99.20:51271/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-12 09:47:11,515 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJsh0qK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJsh0qK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.99.20:51271/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-jq4t.c.gcp-upstream.internal','N') ;"
2026-08-12 09:47:11,743 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJsh0qK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJsh0qK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.99.20:51271/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-12 09:47:12,013 [I] Table x_portal_user does not exist in database ranger
2026-08-12 09:47:12,013 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJsh0qK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJsh0qK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.99.20:51271/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-12 09:47:12,251 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-12 09:47:12,251 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJsh0qK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJsh0qK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.99.20:51271/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-12 09:47:12,502 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJsh0qK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJsh0qK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.99.20:51271/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-12 09:47:12,752 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-12 09:47:12,752 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJsh0qK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJsh0qK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.99.20:51271/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskJsh0qK/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-12 09:47:13.725 UTC [863] WARNING: there is no transaction in progress
2026-08-12 09:47:13.742 UTC [863] WARNING: there is no transaction in progress
2026-08-12 09:47:13.760 UTC [863] WARNING: there is no transaction in progress
2026-08-12 09:47:13.776 UTC [863] WARNING: there is no transaction in progress
2026-08-12 09:47:13.793 UTC [863] WARNING: there is no transaction in progress
2026-08-12 09:47:13.811 UTC [863] WARNING: there is no transaction in progress
2026-08-12 09:47:13.875 UTC [863] WARNING: there is no transaction in progress
2026-08-12 09:47:13.882 UTC [863] WARNING: there is no transaction in progress
2026-08-12 09:47:13.897 UTC [863] WARNING: there is no transaction in progress
2026-08-12 09:47:13.907 UTC [863] WARNING: there is no transaction in progress
2026-08-12 09:47:13.918 UTC [863] WARNING: there is no transaction in progress
2026-08-12 09:47:13.933 UTC [863] WARNING: there is no transaction in progress
2026-08-12 09:47:13.942 UTC [863] WARNING: there is no transaction in progress
2026-08-12 09:47:13.951 UTC [863] WARNING: there is no transaction in progress
2026-08-12 09:47:13.959 UTC [863] WARNING: there is no transaction in progress
2026-08-12 09:47:13.967 UTC [863] WARNING: there is no transaction in progress
2026-08-12 09:47:14,673 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-12 09:47:14,673 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJsh0qK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJsh0qK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.99.20:51271/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-12 09:47:14,925 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-12 09:47:14,926 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJsh0qK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJsh0qK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.99.20:51271/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-12 09:47:15,175 [I] Table x_portal_user already exists in database 'ranger'
2026-08-12 09:47:15,175 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJsh0qK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJsh0qK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.99.20:51271/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-12 09:47:15,417 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-12 09:47:15,418 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJsh0qK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJsh0qK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.99.20:51271/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-12 09:47:15,652 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJsh0qK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJsh0qK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.99.20:51271/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-12 09:47:15,919 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJsh0qK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJsh0qK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.99.20:51271/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-jq4t.c.gcp-upstream.internal';"
2026-08-12 09:47:16,148 [I] CORE_DB_SCHEMA import status has been updated
2026-08-12 09:47:16,149 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJsh0qK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJsh0qK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.99.20:51271/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-12 09:47:16,419 [I] DB_PATCHES have already been applied
I20260812 09:47:16.425006 13708 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ranger-admin:/tmp/dist-test-taskJsh0qK/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskJsh0qK/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskJsh0qK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJsh0qK/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260812 09:47:16.425078 13708 mini_ranger.cc:193] Using host: 127.13.99.20
I20260812 09:47:16.429659 13708 mini_ranger.cc:227] Ranger admin URL: http://127.13.99.20:49901
Aug 12, 2026 9:47:16 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 12, 2026 9:47:16 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 12, 2026 9:47:16 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ranger-admin/ews/webapp
Aug 12, 2026 9:47:16 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 12, 2026 9:47:16 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 12, 2026 9:47:17 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ranger-admin/ews/webapp].
Aug 12, 2026 9:47:17 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260812 09:47:17.841945 13708 mini_ranger.cc:160] Time spent starting Ranger: real 8.953s user 0.005s sys 0.007s
Aug 12, 2026 9:47:37 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 12, 2026 9:47:37 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.MetricsREST
Aug 12, 2026 9:47:37 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 12, 2026 9:47:37 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 12, 2026 9:47:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 12, 2026 9:47:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 12, 2026 9:47:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 12, 2026 9:47:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 12, 2026 9:47:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 12, 2026 9:47:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 12, 2026 9:47:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 12, 2026 9:47:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 12, 2026 9:47:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 12, 2026 9:47:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 12, 2026 9:47:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 12, 2026 9:47:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 12, 2026 9:47:37 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 12, 2026 9:47:38 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260812 09:47:38.777258 13708 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.13.99.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.13.99.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.13.99.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/krb5kdc/rangerkms_127.13.99.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.13.99.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/krb5kdc/rangerkms_127.13.99.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.13.99.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.13.99.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.99.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/krb5kdc/HTTP_127.13.99.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.13.99.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/krb5kdc/HTTP_127.13.99.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 12 09:47:38 dist-test-slave-jq4t krb5kdc[475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528058, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260812 09:47:38.866278 13708 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260812 09:47:38.876741 13708 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260812 09:47:39.783061 13708 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260812 09:47:39.783156 13708 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ranger-kms
I20260812 09:47:39.783182 13708 mini_ranger_kms.cc:227] Using postgres at 127.13.99.20:51271
2026-08-12 09:47:39,959 [I] DB FLAVOR :POSTGRES
2026-08-12 09:47:39,960 [I] --------- Verifying Ranger DB connection ---------
2026-08-12 09:47:39,960 [I] Checking connection
2026-08-12 09:47:39,960 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJsh0qK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.99.20:51271/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-12 09:47:40,228 [I] connection success
2026-08-12 09:47:40,228 [I] --------- Verifying Ranger DB tables ---------
2026-08-12 09:47:40,228 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-12 09:47:40,228 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJsh0qK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.99.20:51271/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-12 09:47:40,469 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-12 09:47:40,469 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-12 09:47:40,469 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-12 09:47:40,469 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJsh0qK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.99.20:51271/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-12 09:47:40,740 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260812 09:47:41.241765 13708 mini_ranger_kms.cc:310] Created kms service
I20260812 09:47:41.435741 13708 mini_ranger_kms.cc:326] Created kudu user
I20260812 09:47:41.478802 13708 mini_ranger_kms.cc:343] Created rangerkms user
Aug 12 09:47:41 dist-test-slave-jq4t krb5kdc[475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528058, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.13.99.20@KRBTEST.COM
I20260812 09:47:41.877599 13708 mini_ranger_kms.cc:384] Added ranger policy
I20260812 09:47:41.877760 13708 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ranger-kms:/tmp/dist-test-taskJsh0qK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJsh0qK/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskJsh0qK/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260812 09:47:41.877787 13708 mini_ranger_kms.cc:240] Using host: 127.13.99.20
I20260812 09:47:41.883929 13708 mini_ranger_kms.cc:276] Ranger KMS PID: 1164
I20260812 09:47:41.884020 13708 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.13.99.20:38493
Aug 12, 2026 9:47:43 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 12, 2026 9:47:43 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 12, 2026 9:47:43 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 12, 2026 9:47:43 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 12, 2026 9:47:43 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ranger-kms/ews/webapp].
Aug 12, 2026 9:47:43 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 12 09:47:44 dist-test-slave-jq4t krb5kdc[475](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786528064, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.13.99.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:47:44.667484 13708 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.801s user 0.011s sys 0.151s
I20260812 09:47:44.667634 13708 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260812 09:47:44.667707 13708 mini_ranger_kms.cc:401] 127.13.99.20:38493/kms/v1/keys
Aug 12 09:47:44 dist-test-slave-jq4t krb5kdc[475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528064, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.13.99.20@KRBTEST.COM for HTTP/127.13.99.20@KRBTEST.COM
Aug 12 09:47:45 dist-test-slave-jq4t krb5kdc[475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528064, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.13.99.20@KRBTEST.COM for HTTP/127.13.99.20@KRBTEST.COM
Aug 12 09:47:47 dist-test-slave-jq4t krb5kdc[475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528067, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.13.99.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.99.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.99.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab.
Entry for principal kudu/127.13.99.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab.
I20260812 09:47:47.573338 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.62:36325
--webserver_interface=127.13.99.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab
--principal=kudu/127.13.99.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:37747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.99.62:36325
--ranger_config_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.99.20:38493/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:47:47.703967 1262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:47:47.704375 1262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:47:47.704452 1262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:47:47.708709 1262 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 09:47:47.708788 1262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:47:47.708824 1262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:47:47.708863 1262 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 09:47:47.708896 1262 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 09:47:47.714558 1262 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:37747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/master-0/wal
--ranger_kms_url=127.13.99.20:38493/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.13.99.62:36325
--ranger_config_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab
--principal=kudu/127.13.99.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.62:36325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.99.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1262
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:47:47.715675 1262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:47:47.716827 1262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:47:47.726027 1271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:47:47.726032 1275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:47:47.726034 1272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:47:47.726408 1262 server_base.cc:1034] running on GCE node
I20260812 09:47:47.727025 1262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:47:47.728165 1262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:47:47.729355 1262 hybrid_clock.cc:648] HybridClock initialized: now 1786528067729334 us; error 37 us; skew 500 ppm
Aug 12 09:47:47 dist-test-slave-jq4t krb5kdc[475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528067, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:47:47.732416 1262 init.cc:377] Logged in from keytab as kudu/127.13.99.62@KRBTEST.COM (short username kudu)
I20260812 09:47:47.733490 1262 webserver.cc:460] Webserver started at http://127.13.99.62:39825/ using document root <none> and password file <none>
I20260812 09:47:47.734041 1262 fs_manager.cc:359] Metadata directory not provided
I20260812 09:47:47.734097 1262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:47:47.734318 1262 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 09:47:47 dist-test-slave-jq4t krb5kdc[475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528067, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.62@KRBTEST.COM for HTTP/127.13.99.20@KRBTEST.COM
I20260812 09:47:47.791553 1262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/master-0/data/instance:
uuid: "a504c7abdfd44aaebb24958ae0f857e4"
format_stamp: "Formatted at 2026-08-12 09:47:47 on dist-test-slave-jq4t"
server_key: "612bf12066b0820e2e207a26a15b660f"
server_key_iv: "44490c6ddfcd95f8ec3b364362300df8"
server_key_version: "kuduclusterkey@0"
I20260812 09:47:47.792214 1262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/master-0/wal/instance:
uuid: "a504c7abdfd44aaebb24958ae0f857e4"
format_stamp: "Formatted at 2026-08-12 09:47:47 on dist-test-slave-jq4t"
server_key: "612bf12066b0820e2e207a26a15b660f"
server_key_iv: "44490c6ddfcd95f8ec3b364362300df8"
server_key_version: "kuduclusterkey@0"
I20260812 09:47:47.796435 1262 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260812 09:47:47.812829 1289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:47:47.814504 1262 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 09:47:47.814672 1262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/master-0/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/master-0/wal
uuid: "a504c7abdfd44aaebb24958ae0f857e4"
format_stamp: "Formatted at 2026-08-12 09:47:47 on dist-test-slave-jq4t"
server_key: "612bf12066b0820e2e207a26a15b660f"
server_key_iv: "44490c6ddfcd95f8ec3b364362300df8"
server_key_version: "kuduclusterkey@0"
I20260812 09:47:47.814831 1262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:47:47.830634 1262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:47:47.833914 1262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:47:47.834115 1262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:47:47.842664 1262 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.62:36325
I20260812 09:47:47.842695 1389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.62:36325 every 8 connection(s)
I20260812 09:47:47.843850 1262 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
I20260812 09:47:47.846783 1393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:47:47.852329 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 1262
I20260812 09:47:47.852466 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/master-0/wal/instance
I20260812 09:47:47.852698 1393 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a504c7abdfd44aaebb24958ae0f857e4: Bootstrap starting.
I20260812 09:47:47.852805 13708 external_mini_cluster.cc:1428] Setting key 4b01db0a4c9aa824040a500c8b714c25
I20260812 09:47:47.855366 1393 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a504c7abdfd44aaebb24958ae0f857e4: Neither blocks nor log segments found. Creating new log.
I20260812 09:47:47.856124 1393 log.cc:824] T 00000000000000000000000000000000 P a504c7abdfd44aaebb24958ae0f857e4: Log is configured to *not* fsync() on all Append() calls
I20260812 09:47:47.858250 1393 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a504c7abdfd44aaebb24958ae0f857e4: No bootstrap required, opened a new log
I20260812 09:47:47.862648 1393 raft_consensus.cc:348] T 00000000000000000000000000000000 P a504c7abdfd44aaebb24958ae0f857e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a504c7abdfd44aaebb24958ae0f857e4" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 36325 } }
I20260812 09:47:47.862963 1393 raft_consensus.cc:374] T 00000000000000000000000000000000 P a504c7abdfd44aaebb24958ae0f857e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:47:47.863065 1393 raft_consensus.cc:729] T 00000000000000000000000000000000 P a504c7abdfd44aaebb24958ae0f857e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a504c7abdfd44aaebb24958ae0f857e4, State: Initialized, Role: FOLLOWER
Aug 12 09:47:47 dist-test-slave-jq4t krb5kdc[475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528067, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
I20260812 09:47:47.863615 1393 consensus_queue.cc:260] T 00000000000000000000000000000000 P a504c7abdfd44aaebb24958ae0f857e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a504c7abdfd44aaebb24958ae0f857e4" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 36325 } }
I20260812 09:47:47.863780 1393 raft_consensus.cc:388] T 00000000000000000000000000000000 P a504c7abdfd44aaebb24958ae0f857e4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 09:47:47.863876 1393 raft_consensus.cc:482] T 00000000000000000000000000000000 P a504c7abdfd44aaebb24958ae0f857e4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 09:47:47.864015 1393 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a504c7abdfd44aaebb24958ae0f857e4 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:47:47.865327 1393 raft_consensus.cc:504] T 00000000000000000000000000000000 P a504c7abdfd44aaebb24958ae0f857e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a504c7abdfd44aaebb24958ae0f857e4" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 36325 } }
I20260812 09:47:47.865733 1393 leader_election.cc:302] T 00000000000000000000000000000000 P a504c7abdfd44aaebb24958ae0f857e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a504c7abdfd44aaebb24958ae0f857e4; no voters:
I20260812 09:47:47.866099 1393 leader_election.cc:288] T 00000000000000000000000000000000 P a504c7abdfd44aaebb24958ae0f857e4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 09:47:47.866251 1400 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a504c7abdfd44aaebb24958ae0f857e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 09:47:47.866544 1400 raft_consensus.cc:686] T 00000000000000000000000000000000 P a504c7abdfd44aaebb24958ae0f857e4 [term 1 LEADER]: Becoming Leader. State: Replica: a504c7abdfd44aaebb24958ae0f857e4, State: Running, Role: LEADER
I20260812 09:47:47.866902 1400 consensus_queue.cc:237] T 00000000000000000000000000000000 P a504c7abdfd44aaebb24958ae0f857e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a504c7abdfd44aaebb24958ae0f857e4" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 36325 } }
I20260812 09:47:47.867609 1393 sys_catalog.cc:564] T 00000000000000000000000000000000 P a504c7abdfd44aaebb24958ae0f857e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 09:47:47.868212 1400 sys_catalog.cc:455] T 00000000000000000000000000000000 P a504c7abdfd44aaebb24958ae0f857e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a504c7abdfd44aaebb24958ae0f857e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a504c7abdfd44aaebb24958ae0f857e4" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 36325 } } }
I20260812 09:47:47.868403 1400 sys_catalog.cc:458] T 00000000000000000000000000000000 P a504c7abdfd44aaebb24958ae0f857e4 [sys.catalog]: This master's current role is: LEADER
I20260812 09:47:47.868870 1400 sys_catalog.cc:455] T 00000000000000000000000000000000 P a504c7abdfd44aaebb24958ae0f857e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a504c7abdfd44aaebb24958ae0f857e4. Latest consensus state: current_term: 1 leader_uuid: "a504c7abdfd44aaebb24958ae0f857e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a504c7abdfd44aaebb24958ae0f857e4" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 36325 } } }
I20260812 09:47:47.869022 1400 sys_catalog.cc:458] T 00000000000000000000000000000000 P a504c7abdfd44aaebb24958ae0f857e4 [sys.catalog]: This master's current role is: LEADER
I20260812 09:47:47.869364 1420 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 09:47:47.870443 1397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:47:47.854250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36578 (local address 127.13.99.62:36325)
0812 09:47:47.854771 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0812 09:47:47.854791 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0812 09:47:47.854836 (+ 45us) server_negotiation.cc:407] Connection header received
0812 09:47:47.855683 (+ 847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:47:47.855709 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:47:47.856150 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:47:47.856591 (+ 441us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:47:47.857504 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:47:47.858832 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:47:47.860148 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:47:47.860880 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:47:47.861044 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:47:47.864117 (+ 3073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:47:47.864125 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:47:47.864144 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:47:47.864176 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:47:47.866265 (+ 2089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:47:47.866866 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:47:47.866871 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:47:47.866877 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:47:47.867032 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:47:47.867365 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:47:47.867367 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:47:47.867369 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:47:47.867901 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:47:47.868793 (+ 892us) server_negotiation.cc:1035] Waiting for connection context
0812 09:47:47.868961 (+ 168us) server_negotiation.cc:299] Negotiation successful
0812 09:47:47.869257 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":158,"threads_started":1}
I20260812 09:47:47.873031 1420 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 09:47:47.873485 1393 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260812 09:47:47.884634 1420 catalog_manager.cc:1347] Generated new cluster ID: 0a9652a0efdf4ac9bc929ff94b431fe0
I20260812 09:47:47.884720 1420 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 09:47:47.898567 1420 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 09:47:47.899850 1420 catalog_manager.cc:1504] Loading token signing keys...
I20260812 09:47:47.911751 1420 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a504c7abdfd44aaebb24958ae0f857e4: Generated new TSK 0
I20260812 09:47:47.912578 1420 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.99.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.99.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.99.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.1.keytab.
Entry for principal kudu/127.13.99.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.1.keytab.
I20260812 09:47:48.640514 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.99.20:38493/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.99.1:0
--local_ip_for_outbound_sockets=127.13.99.1
--webserver_interface=127.13.99.1
--webserver_port=0
--tserver_master_addrs=127.13.99.62:36325
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.1.keytab
--principal=kudu/127.13.99.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:37747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:47:48.769513 1479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:47:48.769819 1479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:47:48.769897 1479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:47:48.774364 1479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:47:48.774451 1479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:47:48.774556 1479 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.99.1
I20260812 09:47:48.780460 1479 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:37747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.13.99.20:38493/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.1.keytab
--principal=kudu/127.13.99.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.99.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.99.1
--webserver_port=0
--tserver_master_addrs=127.13.99.62:36325
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1479
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.99.1
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:47:48.781649 1479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:47:48.782758 1479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:47:48.793656 1488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:47:48.793656 1489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:47:48.794019 1479 server_base.cc:1034] running on GCE node
W20260812 09:47:48.794270 1491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:47:48.794678 1479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:47:48.795336 1479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:47:48.796566 1479 hybrid_clock.cc:648] HybridClock initialized: now 1786528068796537 us; error 44 us; skew 500 ppm
Aug 12 09:47:48 dist-test-slave-jq4t krb5kdc[475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528068, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:47:48.799465 1479 init.cc:377] Logged in from keytab as kudu/127.13.99.1@KRBTEST.COM (short username kudu)
I20260812 09:47:48.800369 1479 webserver.cc:460] Webserver started at http://127.13.99.1:35271/ using document root <none> and password file <none>
I20260812 09:47:48.800959 1479 fs_manager.cc:359] Metadata directory not provided
I20260812 09:47:48.801031 1479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:47:48.801254 1479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 09:47:48 dist-test-slave-jq4t krb5kdc[475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528068, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.1@KRBTEST.COM for HTTP/127.13.99.20@KRBTEST.COM
I20260812 09:47:48.817305 1479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ts-0/data/instance:
uuid: "bd25d8d5808244d286f3dd6062abb17e"
format_stamp: "Formatted at 2026-08-12 09:47:48 on dist-test-slave-jq4t"
server_key: "ae855cc1a14a7a3f3eb302de67ac1d97"
server_key_iv: "9d637bc2f1673e00495da343a848c0c4"
server_key_version: "kuduclusterkey@0"
I20260812 09:47:48.817914 1479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ts-0/wal/instance:
uuid: "bd25d8d5808244d286f3dd6062abb17e"
format_stamp: "Formatted at 2026-08-12 09:47:48 on dist-test-slave-jq4t"
server_key: "ae855cc1a14a7a3f3eb302de67ac1d97"
server_key_iv: "9d637bc2f1673e00495da343a848c0c4"
server_key_version: "kuduclusterkey@0"
I20260812 09:47:48.822619 1479 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 09:47:48.840565 1513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:47:48.842201 1479 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 09:47:48.842339 1479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ts-0/wal
uuid: "bd25d8d5808244d286f3dd6062abb17e"
format_stamp: "Formatted at 2026-08-12 09:47:48 on dist-test-slave-jq4t"
server_key: "ae855cc1a14a7a3f3eb302de67ac1d97"
server_key_iv: "9d637bc2f1673e00495da343a848c0c4"
server_key_version: "kuduclusterkey@0"
I20260812 09:47:48.842473 1479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:47:48.858940 1479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:47:48.862129 1479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:47:48.862344 1479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:47:48.863054 1479 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 09:47:48.864180 1479 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:47:48.864288 1479 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 09:47:48.864442 1479 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:47:48.864569 1479 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 09:47:48.876094 1479 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.1:33427
I20260812 09:47:48.876632 1708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.1:33427 every 8 connection(s)
I20260812 09:47:48.877251 1479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ts-0/data/info.pb
I20260812 09:47:48.877648 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 1479
I20260812 09:47:48.877785 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ts-0/wal/instance
I20260812 09:47:48.878142 13708 external_mini_cluster.cc:1428] Setting key 84af76eb8b605015149928f44d8637bd
WARNING: no policy specified for kudu/127.13.99.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.99.2@KRBTEST.COM" created.
Aug 12 09:47:48 dist-test-slave-jq4t krb5kdc[475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528068, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.1@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
I20260812 09:47:48.901152 1717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:47:48.885625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.1:49365 (local address 127.13.99.62:36325)
0812 09:47:48.885966 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0812 09:47:48.885977 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 09:47:48.887411 (+ 1434us) server_negotiation.cc:407] Connection header received
0812 09:47:48.888683 (+ 1272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:47:48.888689 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:47:48.888753 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:47:48.888819 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:47:48.890568 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:47:48.891210 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:47:48.892960 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:47:48.893316 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:47:48.893444 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:47:48.896221 (+ 2777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:47:48.896228 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:47:48.896233 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:47:48.896267 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:47:48.898249 (+ 1982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:47:48.898971 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:47:48.898976 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:47:48.898981 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:47:48.899084 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:47:48.899537 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:47:48.899541 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:47:48.899543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:47:48.899808 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:47:48.899896 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0812 09:47:48.900741 (+ 845us) server_negotiation.cc:299] Negotiation successful
0812 09:47:48.900898 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":128,"threads_started":1}
I20260812 09:47:48.902837 1719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:47:48.886169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:36325 (local address 127.13.99.1:49365)
0812 09:47:48.887186 (+ 1017us) negotiation.cc:107] Waiting for socket to connect
0812 09:47:48.887250 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0812 09:47:48.888387 (+ 1137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:47:48.889278 (+ 891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:47:48.889287 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:47:48.889652 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:47:48.890376 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:47:48.890399 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:47:48.891354 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:47:48.891358 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:47:48.892817 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:47:48.892830 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:47:48.893487 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:47:48.893492 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:47:48.893701 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:47:48.894329 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:47:48.894346 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:47:48.895985 (+ 1639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:47:48.898392 (+ 2407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:47:48.898402 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:47:48.898421 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:47:48.898840 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:47:48.899203 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:47:48.899209 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:47:48.899212 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:47:48.899402 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:47:48.899996 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:47:48.900004 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:47:48.900374 (+ 370us) client_negotiation.cc:769] Sending connection context
0812 09:47:48.901347 (+ 973us) client_negotiation.cc:240] Negotiation successful
0812 09:47:48.901715 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":771,"thread_start_us":153,"threads_started":1}
I20260812 09:47:48.904258 1710 heartbeater.cc:347] Connected to a master server at 127.13.99.62:36325
I20260812 09:47:48.904597 1710 heartbeater.cc:464] Registering TS with master...
I20260812 09:47:48.905124 1710 heartbeater.cc:511] Master 127.13.99.62:36325 requested a full tablet report, sending...
I20260812 09:47:48.906965 1326 ts_manager.cc:194] Registered new tserver with Master: bd25d8d5808244d286f3dd6062abb17e (127.13.99.1:33427)
I20260812 09:47:48.908630 1326 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.99.1@KRBTEST.COM'} at 127.13.99.1:49365
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.99.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.2.keytab.
Entry for principal kudu/127.13.99.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.2.keytab.
I20260812 09:47:48.920163 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.99.20:38493/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.99.2:0
--local_ip_for_outbound_sockets=127.13.99.2
--webserver_interface=127.13.99.2
--webserver_port=0
--tserver_master_addrs=127.13.99.62:36325
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.2.keytab
--principal=kudu/127.13.99.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:37747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 09:47:49 dist-test-slave-jq4t krb5kdc[475](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786528069, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260812 09:47:49.066555 1721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:47:49.066859 1721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:47:49.066933 1721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:47:49.071441 1721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:47:49.071528 1721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:47:49.071614 1721 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.99.2
I20260812 09:47:49.077320 1721 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:37747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.13.99.20:38493/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.2.keytab
--principal=kudu/127.13.99.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.99.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.99.2
--webserver_port=0
--tserver_master_addrs=127.13.99.62:36325
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1721
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.99.2
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:47:49.078467 1721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:47:49.079828 1721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:47:49.092082 1734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:47:49.092424 1721 server_base.cc:1034] running on GCE node
W20260812 09:47:49.092235 1728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:47:49.092800 1729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:47:49.093353 1721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:47:49.094120 1721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:47:49.095322 1721 hybrid_clock.cc:648] HybridClock initialized: now 1786528069095294 us; error 44 us; skew 500 ppm
Aug 12 09:47:49 dist-test-slave-jq4t krb5kdc[475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528069, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:47:49.098973 1721 init.cc:377] Logged in from keytab as kudu/127.13.99.2@KRBTEST.COM (short username kudu)
I20260812 09:47:49.100083 1721 webserver.cc:460] Webserver started at http://127.13.99.2:43017/ using document root <none> and password file <none>
I20260812 09:47:49.100832 1721 fs_manager.cc:359] Metadata directory not provided
I20260812 09:47:49.100922 1721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:47:49.101171 1721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 09:47:49 dist-test-slave-jq4t krb5kdc[475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528069, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.2@KRBTEST.COM for HTTP/127.13.99.20@KRBTEST.COM
I20260812 09:47:49.117717 1721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ts-1/data/instance:
uuid: "05f1de16c6dd4cbcaf2e3a33fd060f16"
format_stamp: "Formatted at 2026-08-12 09:47:49 on dist-test-slave-jq4t"
server_key: "dec1a6db7d7acdcea7f98874808f2bee"
server_key_iv: "2dcb6efba8a9be50429e6cf150d9bbf0"
server_key_version: "kuduclusterkey@0"
I20260812 09:47:49.118495 1721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ts-1/wal/instance:
uuid: "05f1de16c6dd4cbcaf2e3a33fd060f16"
format_stamp: "Formatted at 2026-08-12 09:47:49 on dist-test-slave-jq4t"
server_key: "dec1a6db7d7acdcea7f98874808f2bee"
server_key_iv: "2dcb6efba8a9be50429e6cf150d9bbf0"
server_key_version: "kuduclusterkey@0"
I20260812 09:47:49.123687 1721 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260812 09:47:49.144132 1753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:47:49.145737 1721 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260812 09:47:49.145871 1721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ts-1/wal
uuid: "05f1de16c6dd4cbcaf2e3a33fd060f16"
format_stamp: "Formatted at 2026-08-12 09:47:49 on dist-test-slave-jq4t"
server_key: "dec1a6db7d7acdcea7f98874808f2bee"
server_key_iv: "2dcb6efba8a9be50429e6cf150d9bbf0"
server_key_version: "kuduclusterkey@0"
I20260812 09:47:49.146016 1721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:47:49.169492 1721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:47:49.173328 1721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:47:49.173548 1721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:47:49.174263 1721 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 09:47:49.175354 1721 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:47:49.175421 1721 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 09:47:49.175477 1721 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:47:49.175500 1721 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 09:47:49.186715 1721 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.2:45277
I20260812 09:47:49.186892 1940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.2:45277 every 8 connection(s)
I20260812 09:47:49.188023 1721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ts-1/data/info.pb
I20260812 09:47:49.197544 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 1721
I20260812 09:47:49.197677 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ts-1/wal/instance
I20260812 09:47:49.197993 13708 external_mini_cluster.cc:1428] Setting key f4eb8cf15750e7e48dd3a25eaaa501c4
Aug 12 09:47:49 dist-test-slave-jq4t krb5kdc[475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528069, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.2@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
I20260812 09:47:49.207115 1717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:47:49.192041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.2:56295 (local address 127.13.99.62:36325)
0812 09:47:49.192179 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0812 09:47:49.192183 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:47:49.192688 (+ 505us) server_negotiation.cc:407] Connection header received
0812 09:47:49.193903 (+ 1215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:47:49.193907 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:47:49.193968 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:47:49.194088 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:47:49.195531 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:47:49.196372 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:47:49.198744 (+ 2372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:47:49.199103 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:47:49.199192 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:47:49.202391 (+ 3199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:47:49.202396 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:47:49.202399 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:47:49.202431 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:47:49.204339 (+ 1908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:47:49.205067 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:47:49.205071 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:47:49.205073 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:47:49.205167 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:47:49.205626 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:47:49.205629 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:47:49.205631 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:47:49.205880 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:47:49.205975 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0812 09:47:49.206749 (+ 774us) server_negotiation.cc:299] Negotiation successful
0812 09:47:49.206908 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260812 09:47:49.208853 1944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:47:49.190869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:36325 (local address 127.13.99.2:56295)
0812 09:47:49.192500 (+ 1631us) negotiation.cc:107] Waiting for socket to connect
0812 09:47:49.192560 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0812 09:47:49.193638 (+ 1078us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:47:49.194299 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:47:49.194309 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:47:49.194676 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:47:49.195332 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:47:49.195352 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:47:49.196563 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:47:49.196567 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:47:49.198576 (+ 2009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:47:49.198590 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:47:49.199410 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:47:49.199414 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:47:49.199633 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:47:49.200378 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:47:49.200399 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:47:49.202180 (+ 1781us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:47:49.204535 (+ 2355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:47:49.204542 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:47:49.204560 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:47:49.204937 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:47:49.205289 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:47:49.205292 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:47:49.205294 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:47:49.205486 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:47:49.206056 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:47:49.206065 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:47:49.206426 (+ 361us) client_negotiation.cc:769] Sending connection context
0812 09:47:49.207315 (+ 889us) client_negotiation.cc:240] Negotiation successful
0812 09:47:49.207715 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1365,"thread_start_us":154,"threads_started":1}
I20260812 09:47:49.210419 1941 heartbeater.cc:347] Connected to a master server at 127.13.99.62:36325
I20260812 09:47:49.210695 1941 heartbeater.cc:464] Registering TS with master...
I20260812 09:47:49.211391 1941 heartbeater.cc:511] Master 127.13.99.62:36325 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.13.99.3@KRBTEST.COM; defaulting to no policy
I20260812 09:47:49.212759 1326 ts_manager.cc:194] Registered new tserver with Master: 05f1de16c6dd4cbcaf2e3a33fd060f16 (127.13.99.2:45277)
I20260812 09:47:49.213366 1326 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.99.2@KRBTEST.COM'} at 127.13.99.2:56295
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.99.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.99.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.3.keytab.
Entry for principal kudu/127.13.99.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.3.keytab.
I20260812 09:47:49.236449 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.99.20:38493/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.99.3:0
--local_ip_for_outbound_sockets=127.13.99.3
--webserver_interface=127.13.99.3
--webserver_port=0
--tserver_master_addrs=127.13.99.62:36325
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.3.keytab
--principal=kudu/127.13.99.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:37747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:47:49.403959 1951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:47:49.404384 1951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:47:49.404498 1951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:47:49.409960 1951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:47:49.410048 1951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:47:49.410147 1951 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.99.3
I20260812 09:47:49.416967 1951 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:37747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.13.99.20:38493/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.3.keytab
--principal=kudu/127.13.99.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.99.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.99.3
--webserver_port=0
--tserver_master_addrs=127.13.99.62:36325
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1951
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.99.3
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:47:49.418154 1951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:47:49.419274 1951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:47:49.431001 1963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:47:49.431098 1966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:47:49.431000 1964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:47:49.432359 1951 server_base.cc:1034] running on GCE node
I20260812 09:47:49.432852 1951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:47:49.433590 1951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:47:49.434790 1951 hybrid_clock.cc:648] HybridClock initialized: now 1786528069434752 us; error 58 us; skew 500 ppm
Aug 12 09:47:49 dist-test-slave-jq4t krb5kdc[475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528069, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:47:49.438131 1951 init.cc:377] Logged in from keytab as kudu/127.13.99.3@KRBTEST.COM (short username kudu)
I20260812 09:47:49.439172 1951 webserver.cc:460] Webserver started at http://127.13.99.3:43263/ using document root <none> and password file <none>
I20260812 09:47:49.439838 1951 fs_manager.cc:359] Metadata directory not provided
I20260812 09:47:49.439935 1951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:47:49.440165 1951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 09:47:49 dist-test-slave-jq4t krb5kdc[475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528069, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.3@KRBTEST.COM for HTTP/127.13.99.20@KRBTEST.COM
I20260812 09:47:49.454547 1951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ts-2/data/instance:
uuid: "a6bf7dc8e82242c3950f456c5de6a03c"
format_stamp: "Formatted at 2026-08-12 09:47:49 on dist-test-slave-jq4t"
server_key: "e674e3c916bc1c109c745176ce203152"
server_key_iv: "49796a1a026650012db2a5942768660b"
server_key_version: "kuduclusterkey@0"
I20260812 09:47:49.455360 1951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ts-2/wal/instance:
uuid: "a6bf7dc8e82242c3950f456c5de6a03c"
format_stamp: "Formatted at 2026-08-12 09:47:49 on dist-test-slave-jq4t"
server_key: "e674e3c916bc1c109c745176ce203152"
server_key_iv: "49796a1a026650012db2a5942768660b"
server_key_version: "kuduclusterkey@0"
I20260812 09:47:49.461897 1951 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20260812 09:47:49.484093 1979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:47:49.485845 1951 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260812 09:47:49.486002 1951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ts-2/wal
uuid: "a6bf7dc8e82242c3950f456c5de6a03c"
format_stamp: "Formatted at 2026-08-12 09:47:49 on dist-test-slave-jq4t"
server_key: "e674e3c916bc1c109c745176ce203152"
server_key_iv: "49796a1a026650012db2a5942768660b"
server_key_version: "kuduclusterkey@0"
I20260812 09:47:49.486156 1951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Aug 12 09:47:49 dist-test-slave-jq4t krb5kdc[475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528069, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.62@KRBTEST.COM for HTTP/127.13.99.20@KRBTEST.COM
I20260812 09:47:49.544782 1951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:47:49.548883 1951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:47:49.549119 1951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:47:49.549831 1951 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 09:47:49.550969 1951 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:47:49.551033 1951 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 09:47:49.551090 1951 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:47:49.551112 1951 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 09:47:49.571439 1951 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.3:46447
I20260812 09:47:49.571501 2184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.3:46447 every 8 connection(s)
I20260812 09:47:49.572762 1951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ts-2/data/info.pb
I20260812 09:47:49.573463 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 1951
I20260812 09:47:49.573599 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ts-2/wal/instance
I20260812 09:47:49.573912 13708 external_mini_cluster.cc:1428] Setting key cc5ec9e33c96363ab65e7b5ce40a1b78
Aug 12 09:47:49 dist-test-slave-jq4t krb5kdc[475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528069, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.3@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
I20260812 09:47:49.594923 1717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:47:49.580353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.3:50931 (local address 127.13.99.62:36325)
0812 09:47:49.580471 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0812 09:47:49.580502 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0812 09:47:49.580521 (+ 19us) server_negotiation.cc:407] Connection header received
0812 09:47:49.581278 (+ 757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:47:49.581282 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:47:49.581344 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:47:49.581403 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:47:49.583143 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:47:49.584015 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:47:49.585812 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:47:49.586173 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:47:49.586256 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:47:49.590242 (+ 3986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:47:49.590246 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:47:49.590248 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:47:49.590279 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:47:49.592194 (+ 1915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:47:49.592942 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:47:49.592946 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:47:49.592948 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:47:49.593042 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:47:49.593474 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:47:49.593478 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:47:49.593479 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:47:49.593738 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:47:49.593830 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0812 09:47:49.594596 (+ 766us) server_negotiation.cc:299] Negotiation successful
0812 09:47:49.594739 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260812 09:47:49.596911 2189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:47:49.579172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:36325 (local address 127.13.99.3:50931)
0812 09:47:49.579815 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0812 09:47:49.579879 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0812 09:47:49.581003 (+ 1124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:47:49.581827 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:47:49.581837 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:47:49.582250 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:47:49.582950 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:47:49.582975 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:47:49.584187 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:47:49.584190 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:47:49.585662 (+ 1472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:47:49.585674 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:47:49.586487 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:47:49.586491 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:47:49.586694 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:47:49.588127 (+ 1433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:47:49.588149 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:47:49.590038 (+ 1889us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:47:49.592388 (+ 2350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:47:49.592394 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:47:49.592458 (+ 64us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:47:49.592820 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:47:49.593158 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:47:49.593161 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:47:49.593163 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:47:49.593350 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:47:49.593910 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:47:49.593916 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:47:49.594278 (+ 362us) client_negotiation.cc:769] Sending connection context
0812 09:47:49.595400 (+ 1122us) client_negotiation.cc:240] Negotiation successful
0812 09:47:49.595790 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":189,"threads_started":1}
I20260812 09:47:49.599895 2185 heartbeater.cc:347] Connected to a master server at 127.13.99.62:36325
I20260812 09:47:49.600153 2185 heartbeater.cc:464] Registering TS with master...
I20260812 09:47:49.600778 2185 heartbeater.cc:511] Master 127.13.99.62:36325 requested a full tablet report, sending...
I20260812 09:47:49.602373 1326 ts_manager.cc:194] Registered new tserver with Master: a6bf7dc8e82242c3950f456c5de6a03c (127.13.99.3:46447)
I20260812 09:47:49.603012 1326 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.99.3@KRBTEST.COM'} at 127.13.99.3:50931
I20260812 09:47:49.614019 13708 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 09:47:49.614873 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 1479
I20260812 09:47:49.666970 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 1721
I20260812 09:47:49.681859 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 1951
I20260812 09:47:49.699550 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 1262
I20260812 09:47:49.712165 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.62:36325
--webserver_interface=127.13.99.62
--webserver_port=39825
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab
--principal=kudu/127.13.99.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:37747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.99.62:36325
--ranger_config_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.99.20:38493/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.13.99.20:38493/kms with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:47:49.831545 2194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:47:49.831811 2194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:47:49.831851 2194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:47:49.836058 2194 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 09:47:49.836144 2194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:47:49.836166 2194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:47:49.836182 2194 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 09:47:49.836198 2194 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 09:47:49.841948 2194 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:37747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.13.99.20:38493/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.13.99.62:36325
--ranger_config_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab
--principal=kudu/127.13.99.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.62:36325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.99.62
--webserver_port=39825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2194
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:47:49.842983 2194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:47:49.843958 2194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:47:49.852641 2210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:47:49.852654 2204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:47:49.852757 2194 server_base.cc:1034] running on GCE node
W20260812 09:47:49.852653 2206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:47:49.853505 2194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:47:49.854590 2194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:47:49.855757 2194 hybrid_clock.cc:648] HybridClock initialized: now 1786528069855744 us; error 24 us; skew 500 ppm
Aug 12 09:47:49 dist-test-slave-jq4t krb5kdc[475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528069, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:47:49.858785 2194 init.cc:377] Logged in from keytab as kudu/127.13.99.62@KRBTEST.COM (short username kudu)
I20260812 09:47:49.859663 2194 webserver.cc:460] Webserver started at http://127.13.99.62:39825/ using document root <none> and password file <none>
I20260812 09:47:49.860211 2194 fs_manager.cc:359] Metadata directory not provided
I20260812 09:47:49.860282 2194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 12 09:47:49 dist-test-slave-jq4t krb5kdc[475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528069, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.62@KRBTEST.COM for HTTP/127.13.99.20@KRBTEST.COM
I20260812 09:47:49.909844 2194 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 09:47:49.912441 2223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:47:49.913890 2194 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 09:47:49.914086 2194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/master-0/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/master-0/wal
uuid: "a504c7abdfd44aaebb24958ae0f857e4"
format_stamp: "Formatted at 2026-08-12 09:47:47 on dist-test-slave-jq4t"
server_key: "612bf12066b0820e2e207a26a15b660f"
server_key_iv: "44490c6ddfcd95f8ec3b364362300df8"
server_key_version: "kuduclusterkey@0"
I20260812 09:47:49.914656 2194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:47:49.927006 2194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:47:49.929886 2194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:47:49.930097 2194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:47:49.937690 2194 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.62:36325
I20260812 09:47:49.937716 2335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.62:36325 every 8 connection(s)
I20260812 09:47:49.938766 2194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
I20260812 09:47:49.940227 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 2194
I20260812 09:47:49.940749 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.99.20:38493/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.99.1:33427
--local_ip_for_outbound_sockets=127.13.99.1
--tserver_master_addrs=127.13.99.62:36325
--webserver_port=35271
--webserver_interface=127.13.99.1
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.1.keytab
--principal=kudu/127.13.99.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:37747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.13.99.20:38493/kms with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 09:47:49.943297 2336 sys_catalog.cc:263] Verifying existing consensus state
I20260812 09:47:49.945055 2336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a504c7abdfd44aaebb24958ae0f857e4: Bootstrap starting.
I20260812 09:47:49.961658 2336 log.cc:824] T 00000000000000000000000000000000 P a504c7abdfd44aaebb24958ae0f857e4: Log is configured to *not* fsync() on all Append() calls
I20260812 09:47:49.968252 2336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a504c7abdfd44aaebb24958ae0f857e4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 09:47:49.968726 2336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a504c7abdfd44aaebb24958ae0f857e4: Bootstrap complete.
I20260812 09:47:49.973537 2336 raft_consensus.cc:348] T 00000000000000000000000000000000 P a504c7abdfd44aaebb24958ae0f857e4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a504c7abdfd44aaebb24958ae0f857e4" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 36325 } }
I20260812 09:47:49.973959 2336 raft_consensus.cc:729] T 00000000000000000000000000000000 P a504c7abdfd44aaebb24958ae0f857e4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a504c7abdfd44aaebb24958ae0f857e4, State: Initialized, Role: FOLLOWER
I20260812 09:47:49.974462 2336 consensus_queue.cc:260] T 00000000000000000000000000000000 P a504c7abdfd44aaebb24958ae0f857e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a504c7abdfd44aaebb24958ae0f857e4" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 36325 } }
I20260812 09:47:49.974622 2336 raft_consensus.cc:388] T 00000000000000000000000000000000 P a504c7abdfd44aaebb24958ae0f857e4 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 09:47:49.974697 2336 raft_consensus.cc:482] T 00000000000000000000000000000000 P a504c7abdfd44aaebb24958ae0f857e4 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 09:47:49.974797 2336 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a504c7abdfd44aaebb24958ae0f857e4 [term 1 FOLLOWER]: Advancing to term 2
I20260812 09:47:49.976963 2336 raft_consensus.cc:504] T 00000000000000000000000000000000 P a504c7abdfd44aaebb24958ae0f857e4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a504c7abdfd44aaebb24958ae0f857e4" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 36325 } }
I20260812 09:47:49.977298 2336 leader_election.cc:302] T 00000000000000000000000000000000 P a504c7abdfd44aaebb24958ae0f857e4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a504c7abdfd44aaebb24958ae0f857e4; no voters:
I20260812 09:47:49.977640 2336 leader_election.cc:288] T 00000000000000000000000000000000 P a504c7abdfd44aaebb24958ae0f857e4 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260812 09:47:49.977730 2344 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a504c7abdfd44aaebb24958ae0f857e4 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 09:47:49.977929 2344 raft_consensus.cc:686] T 00000000000000000000000000000000 P a504c7abdfd44aaebb24958ae0f857e4 [term 2 LEADER]: Becoming Leader. State: Replica: a504c7abdfd44aaebb24958ae0f857e4, State: Running, Role: LEADER
I20260812 09:47:49.978233 2344 consensus_queue.cc:237] T 00000000000000000000000000000000 P a504c7abdfd44aaebb24958ae0f857e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a504c7abdfd44aaebb24958ae0f857e4" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 36325 } }
I20260812 09:47:49.978896 2336 sys_catalog.cc:564] T 00000000000000000000000000000000 P a504c7abdfd44aaebb24958ae0f857e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 09:47:49.979182 2344 sys_catalog.cc:455] T 00000000000000000000000000000000 P a504c7abdfd44aaebb24958ae0f857e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a504c7abdfd44aaebb24958ae0f857e4. Latest consensus state: current_term: 2 leader_uuid: "a504c7abdfd44aaebb24958ae0f857e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a504c7abdfd44aaebb24958ae0f857e4" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 36325 } } }
I20260812 09:47:49.979305 2344 sys_catalog.cc:458] T 00000000000000000000000000000000 P a504c7abdfd44aaebb24958ae0f857e4 [sys.catalog]: This master's current role is: LEADER
I20260812 09:47:49.979666 2350 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 09:47:49.979702 2345 sys_catalog.cc:455] T 00000000000000000000000000000000 P a504c7abdfd44aaebb24958ae0f857e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a504c7abdfd44aaebb24958ae0f857e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a504c7abdfd44aaebb24958ae0f857e4" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 36325 } } }
I20260812 09:47:49.979914 2345 sys_catalog.cc:458] T 00000000000000000000000000000000 P a504c7abdfd44aaebb24958ae0f857e4 [sys.catalog]: This master's current role is: LEADER
I20260812 09:47:49.983009 2350 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 09:47:49.983615 2336 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260812 09:47:49.983712 2350 catalog_manager.cc:1259] Loaded cluster ID: 0a9652a0efdf4ac9bc929ff94b431fe0
I20260812 09:47:49.986999 2350 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 09:47:49.991145 2350 catalog_manager.cc:1504] Loading token signing keys...
I20260812 09:47:49.991997 2350 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a504c7abdfd44aaebb24958ae0f857e4: Loaded TSK: 0
I20260812 09:47:49.993498 2350 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 09:47:50.088011 2341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:47:50.088368 2341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:47:50.088450 2341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:47:50.094487 2341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:47:50.094568 2341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:47:50.094663 2341 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.99.1
I20260812 09:47:50.102398 2341 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:37747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.13.99.20:38493/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.1.keytab
--principal=kudu/127.13.99.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.99.1:33427
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.99.1
--webserver_port=35271
--tserver_master_addrs=127.13.99.62:36325
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2341
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.99.1
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:47:50.103777 2341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:47:50.105129 2341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:47:50.117352 2404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:47:50.117714 2402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:47:50.118076 2341 server_base.cc:1034] running on GCE node
W20260812 09:47:50.118331 2401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:47:50.118809 2341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:47:50.119611 2341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:47:50.122917 2341 hybrid_clock.cc:648] HybridClock initialized: now 1786528070122844 us; error 83 us; skew 500 ppm
Aug 12 09:47:50 dist-test-slave-jq4t krb5kdc[475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528070, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:47:50.127930 2341 init.cc:377] Logged in from keytab as kudu/127.13.99.1@KRBTEST.COM (short username kudu)
I20260812 09:47:50.129132 2341 webserver.cc:460] Webserver started at http://127.13.99.1:35271/ using document root <none> and password file <none>
I20260812 09:47:50.129868 2341 fs_manager.cc:359] Metadata directory not provided
I20260812 09:47:50.129987 2341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 12 09:47:50 dist-test-slave-jq4t krb5kdc[475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528070, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.1@KRBTEST.COM for HTTP/127.13.99.20@KRBTEST.COM
I20260812 09:47:50.152675 2341 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.003s sys 0.000s
I20260812 09:47:50.156239 2422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:47:50.157833 2341 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 09:47:50.157972 2341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ts-0/wal
uuid: "bd25d8d5808244d286f3dd6062abb17e"
format_stamp: "Formatted at 2026-08-12 09:47:48 on dist-test-slave-jq4t"
server_key: "ae855cc1a14a7a3f3eb302de67ac1d97"
server_key_iv: "9d637bc2f1673e00495da343a848c0c4"
server_key_version: "kuduclusterkey@0"
I20260812 09:47:50.158731 2341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:47:50.187110 2341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:47:50.190975 2341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:47:50.191207 2341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:47:50.192000 2341 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 09:47:50.193068 2341 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:47:50.193132 2341 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 09:47:50.193189 2341 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:47:50.193214 2341 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 09:47:50.209661 2341 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.1:33427
I20260812 09:47:50.210017 2626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.1:33427 every 8 connection(s)
I20260812 09:47:50.211066 2341 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ts-0/data/info.pb
I20260812 09:47:50.212249 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 2341
I20260812 09:47:50.212795 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.99.20:38493/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.99.2:45277
--local_ip_for_outbound_sockets=127.13.99.2
--tserver_master_addrs=127.13.99.62:36325
--webserver_port=43017
--webserver_interface=127.13.99.2
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.2.keytab
--principal=kudu/127.13.99.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:37747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.13.99.20:38493/kms with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 09:47:50 dist-test-slave-jq4t krb5kdc[475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528070, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.1@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
I20260812 09:47:50.250692 2634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:47:50.214424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:36325 (local address 127.13.99.1:35841)
0812 09:47:50.216999 (+ 2575us) negotiation.cc:107] Waiting for socket to connect
0812 09:47:50.217059 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0812 09:47:50.218310 (+ 1251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:47:50.220462 (+ 2152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:47:50.220491 (+ 29us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:47:50.220872 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:47:50.221573 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:47:50.221596 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:47:50.223744 (+ 2148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:47:50.223747 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:47:50.225198 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:47:50.225210 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:47:50.226594 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:47:50.226598 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:47:50.226781 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:47:50.227531 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:47:50.227554 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:47:50.234130 (+ 6576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:47:50.237639 (+ 3509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:47:50.237647 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:47:50.237664 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:47:50.238040 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:47:50.238486 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:47:50.238490 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:47:50.238492 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:47:50.238680 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:47:50.239915 (+ 1235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:47:50.239923 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:47:50.244572 (+ 4649us) client_negotiation.cc:769] Sending connection context
0812 09:47:50.244807 (+ 235us) client_negotiation.cc:240] Negotiation successful
0812 09:47:50.246485 (+ 1678us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2331,"thread_start_us":154,"threads_started":1}
I20260812 09:47:50.251659 2637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:47:50.215981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.1:35841 (local address 127.13.99.62:36325)
0812 09:47:50.216719 (+ 738us) server_negotiation.cc:206] Beginning negotiation
0812 09:47:50.216743 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0812 09:47:50.218057 (+ 1314us) server_negotiation.cc:407] Connection header received
0812 09:47:50.218975 (+ 918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:47:50.218990 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:47:50.219492 (+ 502us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:47:50.219952 (+ 460us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:47:50.221785 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:47:50.223341 (+ 1556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:47:50.225341 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:47:50.226136 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:47:50.226298 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:47:50.234456 (+ 8158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:47:50.234465 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:47:50.234487 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:47:50.234526 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:47:50.237459 (+ 2933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:47:50.238194 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:47:50.238201 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:47:50.238207 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:47:50.238359 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:47:50.238793 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:47:50.238796 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:47:50.238798 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:47:50.239352 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:47:50.239500 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0812 09:47:50.245214 (+ 5714us) server_negotiation.cc:299] Negotiation successful
0812 09:47:50.245549 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":114,"threads_started":1}
I20260812 09:47:50.254031 2627 heartbeater.cc:347] Connected to a master server at 127.13.99.62:36325
I20260812 09:47:50.254316 2627 heartbeater.cc:464] Registering TS with master...
I20260812 09:47:50.254979 2627 heartbeater.cc:511] Master 127.13.99.62:36325 requested a full tablet report, sending...
W20260812 09:47:50.257262 2627 heartbeater.cc:644] Failed to heartbeat to 127.13.99.62:36325 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260812 09:47:50.257402 2627 heartbeater.cc:464] Registering TS with master...
I20260812 09:47:50.257622 2627 heartbeater.cc:511] Master 127.13.99.62:36325 requested a full tablet report, sending...
I20260812 09:47:50.258514 2627 heartbeater.cc:464] Registering TS with master...
I20260812 09:47:50.258714 2627 heartbeater.cc:511] Master 127.13.99.62:36325 requested a full tablet report, sending...
W20260812 09:47:50.259572 2627 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 09:47:50.397679 2632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:47:50.398080 2632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:47:50.398160 2632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:47:50.404601 2632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:47:50.404695 2632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:47:50.404798 2632 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.99.2
I20260812 09:47:50.413061 2632 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:37747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.13.99.20:38493/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.2.keytab
--principal=kudu/127.13.99.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.99.2:45277
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.99.2
--webserver_port=43017
--tserver_master_addrs=127.13.99.62:36325
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2632
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.99.2
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:47:50.414511 2632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:47:50.415967 2632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:47:50.431447 2647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:47:50.432406 2656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:47:50.432760 2651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:47:50.437124 2632 server_base.cc:1034] running on GCE node
I20260812 09:47:50.437610 2632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:47:50.438401 2632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:47:50.439589 2632 hybrid_clock.cc:648] HybridClock initialized: now 1786528070439550 us; error 49 us; skew 500 ppm
Aug 12 09:47:50 dist-test-slave-jq4t krb5kdc[475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528070, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:47:50.442585 2632 init.cc:377] Logged in from keytab as kudu/127.13.99.2@KRBTEST.COM (short username kudu)
I20260812 09:47:50.443622 2632 webserver.cc:460] Webserver started at http://127.13.99.2:43017/ using document root <none> and password file <none>
I20260812 09:47:50.444209 2632 fs_manager.cc:359] Metadata directory not provided
I20260812 09:47:50.444321 2632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 12 09:47:50 dist-test-slave-jq4t krb5kdc[475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528070, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.2@KRBTEST.COM for HTTP/127.13.99.20@KRBTEST.COM
I20260812 09:47:50.473843 2632 fs_manager.cc:709] Time spent opening directory manager: real 0.011s user 0.003s sys 0.000s
I20260812 09:47:50.477844 2669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:47:50.479437 2632 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260812 09:47:50.479624 2632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ts-1/wal
uuid: "05f1de16c6dd4cbcaf2e3a33fd060f16"
format_stamp: "Formatted at 2026-08-12 09:47:49 on dist-test-slave-jq4t"
server_key: "dec1a6db7d7acdcea7f98874808f2bee"
server_key_iv: "2dcb6efba8a9be50429e6cf150d9bbf0"
server_key_version: "kuduclusterkey@0"
I20260812 09:47:50.480461 2632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:47:50.529947 2632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:47:50.533479 2632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:47:50.533715 2632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:47:50.534503 2632 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 09:47:50.535657 2632 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:47:50.535733 2632 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 09:47:50.535804 2632 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:47:50.535837 2632 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 09:47:50.548418 2632 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.2:45277
I20260812 09:47:50.548843 2879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.2:45277 every 8 connection(s)
I20260812 09:47:50.549852 2632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ts-1/data/info.pb
I20260812 09:47:50.560046 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 2632
I20260812 09:47:50.560683 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.99.20:38493/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.99.3:46447
--local_ip_for_outbound_sockets=127.13.99.3
--tserver_master_addrs=127.13.99.62:36325
--webserver_port=43263
--webserver_interface=127.13.99.3
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.3.keytab
--principal=kudu/127.13.99.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:37747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.13.99.20:38493/kms with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 09:47:50 dist-test-slave-jq4t krb5kdc[475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528070, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.2@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
I20260812 09:47:50.565714 2637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:47:50.552547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.2:44773 (local address 127.13.99.62:36325)
0812 09:47:50.552688 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0812 09:47:50.552692 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:47:50.553720 (+ 1028us) server_negotiation.cc:407] Connection header received
0812 09:47:50.554656 (+ 936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:47:50.554660 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:47:50.554718 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:47:50.554830 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:47:50.556128 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:47:50.556766 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:47:50.557959 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:47:50.558191 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:47:50.558428 (+ 237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:47:50.561603 (+ 3175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:47:50.561610 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:47:50.561617 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:47:50.561656 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:47:50.563297 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:47:50.563921 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:47:50.563925 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:47:50.563927 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:47:50.564011 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:47:50.564385 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:47:50.564389 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:47:50.564390 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:47:50.564644 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:47:50.564762 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0812 09:47:50.565410 (+ 648us) server_negotiation.cc:299] Negotiation successful
0812 09:47:50.565543 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260812 09:47:50.566836 2884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:47:50.553041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:36325 (local address 127.13.99.2:44773)
0812 09:47:50.553568 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0812 09:47:50.553613 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 09:47:50.554449 (+ 836us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:47:50.555023 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:47:50.555030 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:47:50.555384 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:47:50.555977 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:47:50.555992 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:47:50.556903 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:47:50.556906 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:47:50.557822 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:47:50.557832 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:47:50.558463 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:47:50.558467 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:47:50.558648 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:47:50.559675 (+ 1027us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:47:50.559693 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:47:50.561391 (+ 1698us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:47:50.563426 (+ 2035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:47:50.563432 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:47:50.563447 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:47:50.563829 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:47:50.564140 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:47:50.564143 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:47:50.564144 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:47:50.564278 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:47:50.564803 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:47:50.564808 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:47:50.565165 (+ 357us) client_negotiation.cc:769] Sending connection context
0812 09:47:50.565361 (+ 196us) client_negotiation.cc:240] Negotiation successful
0812 09:47:50.565680 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":141,"threads_started":1}
I20260812 09:47:50.568228 2882 heartbeater.cc:347] Connected to a master server at 127.13.99.62:36325
I20260812 09:47:50.568547 2882 heartbeater.cc:464] Registering TS with master...
I20260812 09:47:50.569141 2882 heartbeater.cc:511] Master 127.13.99.62:36325 requested a full tablet report, sending...
W20260812 09:47:50.570916 2882 heartbeater.cc:644] Failed to heartbeat to 127.13.99.62:36325 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260812 09:47:50.571053 2882 heartbeater.cc:464] Registering TS with master...
I20260812 09:47:50.571249 2882 heartbeater.cc:511] Master 127.13.99.62:36325 requested a full tablet report, sending...
I20260812 09:47:50.572108 2882 heartbeater.cc:464] Registering TS with master...
I20260812 09:47:50.572290 2882 heartbeater.cc:511] Master 127.13.99.62:36325 requested a full tablet report, sending...
W20260812 09:47:50.573051 2882 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 09:47:50.725227 2885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:47:50.725636 2885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:47:50.725718 2885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:47:50.730309 2885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:47:50.730402 2885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:47:50.730501 2885 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.99.3
I20260812 09:47:50.737444 2885 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:37747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.13.99.20:38493/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.3.keytab
--principal=kudu/127.13.99.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.99.3:46447
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.99.3
--webserver_port=43263
--tserver_master_addrs=127.13.99.62:36325
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2885
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.99.3
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:47:50.738688 2885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:47:50.739923 2885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:47:50.752271 2894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:47:50.752272 2898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:47:50.753314 2893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:47:50.753345 2885 server_base.cc:1034] running on GCE node
I20260812 09:47:50.753973 2885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:47:50.754799 2885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:47:50.758116 2885 hybrid_clock.cc:648] HybridClock initialized: now 1786528070758073 us; error 55 us; skew 500 ppm
Aug 12 09:47:50 dist-test-slave-jq4t krb5kdc[475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528070, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:47:50.761978 2885 init.cc:377] Logged in from keytab as kudu/127.13.99.3@KRBTEST.COM (short username kudu)
I20260812 09:47:50.763283 2885 webserver.cc:460] Webserver started at http://127.13.99.3:43263/ using document root <none> and password file <none>
I20260812 09:47:50.764232 2885 fs_manager.cc:359] Metadata directory not provided
I20260812 09:47:50.764448 2885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 12 09:47:50 dist-test-slave-jq4t krb5kdc[475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528070, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.3@KRBTEST.COM for HTTP/127.13.99.20@KRBTEST.COM
I20260812 09:47:50.781168 2885 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260812 09:47:50.783999 2913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:47:50.785563 2885 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 09:47:50.785719 2885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ts-2/wal
uuid: "a6bf7dc8e82242c3950f456c5de6a03c"
format_stamp: "Formatted at 2026-08-12 09:47:49 on dist-test-slave-jq4t"
server_key: "e674e3c916bc1c109c745176ce203152"
server_key_iv: "49796a1a026650012db2a5942768660b"
server_key_version: "kuduclusterkey@0"
I20260812 09:47:50.786294 2885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:47:50.804963 2885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:47:50.808133 2885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:47:50.808336 2885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:47:50.809052 2885 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 09:47:50.810004 2885 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:47:50.810053 2885 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 09:47:50.810094 2885 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:47:50.810148 2885 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 09:47:50.821492 2885 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.3:46447
I20260812 09:47:50.821655 3145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.3:46447 every 8 connection(s)
I20260812 09:47:50.822655 2885 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ts-2/data/info.pb
I20260812 09:47:50.831308 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 2885
Aug 12 09:47:50 dist-test-slave-jq4t krb5kdc[475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528070, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.3@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
I20260812 09:47:50.841410 2637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:47:50.825031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.3:35719 (local address 127.13.99.62:36325)
0812 09:47:50.825196 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0812 09:47:50.825201 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:47:50.826163 (+ 962us) server_negotiation.cc:407] Connection header received
0812 09:47:50.827388 (+ 1225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:47:50.827391 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:47:50.827443 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:47:50.827531 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:47:50.829066 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:47:50.829892 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:47:50.831783 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:47:50.832159 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:47:50.832248 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:47:50.836916 (+ 4668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:47:50.836920 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:47:50.836923 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:47:50.836964 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:47:50.838787 (+ 1823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:47:50.839440 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:47:50.839443 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:47:50.839445 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:47:50.839556 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:47:50.839989 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:47:50.839993 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:47:50.839995 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:47:50.840217 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:47:50.840290 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0812 09:47:50.841082 (+ 792us) server_negotiation.cc:299] Negotiation successful
0812 09:47:50.841229 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62,"spinlock_wait_cycles":3840}
I20260812 09:47:50.843161 3154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:47:50.825454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:36325 (local address 127.13.99.3:35719)
0812 09:47:50.825977 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0812 09:47:50.826035 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0812 09:47:50.827112 (+ 1077us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:47:50.827754 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:47:50.827764 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:47:50.828122 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:47:50.828882 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:47:50.828903 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:47:50.830066 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:47:50.830070 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:47:50.831622 (+ 1552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:47:50.831635 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:47:50.832505 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:47:50.832511 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:47:50.832733 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:47:50.834374 (+ 1641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:47:50.834396 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:47:50.836708 (+ 2312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:47:50.838927 (+ 2219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:47:50.838933 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:47:50.838952 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:47:50.839318 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:47:50.839671 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:47:50.839674 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:47:50.839676 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:47:50.839860 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:47:50.840378 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:47:50.840385 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:47:50.840775 (+ 390us) client_negotiation.cc:769] Sending connection context
0812 09:47:50.841650 (+ 875us) client_negotiation.cc:240] Negotiation successful
0812 09:47:50.842022 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":118,"threads_started":1}
I20260812 09:47:50.844439 3156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:47:50.832016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36584 (local address 127.13.99.62:36325)
0812 09:47:50.832697 (+ 681us) server_negotiation.cc:206] Beginning negotiation
0812 09:47:50.832707 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 09:47:50.832987 (+ 280us) server_negotiation.cc:407] Connection header received
0812 09:47:50.833211 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:47:50.833214 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:47:50.833270 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:47:50.833375 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:47:50.834296 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:47:50.835117 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:47:50.836375 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:47:50.836777 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:47:50.836879 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:47:50.839341 (+ 2462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:47:50.839344 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:47:50.839347 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:47:50.839827 (+ 480us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:47:50.842014 (+ 2187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:47:50.842551 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:47:50.842554 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:47:50.842556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:47:50.842649 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:47:50.842975 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:47:50.842979 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:47:50.842981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:47:50.843189 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:47:50.843710 (+ 521us) server_negotiation.cc:1035] Waiting for connection context
0812 09:47:50.844142 (+ 432us) server_negotiation.cc:299] Negotiation successful
0812 09:47:50.844269 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":130,"threads_started":1}
I20260812 09:47:50.844650 3149 heartbeater.cc:347] Connected to a master server at 127.13.99.62:36325
I20260812 09:47:50.844902 3149 heartbeater.cc:464] Registering TS with master...
I20260812 09:47:50.845422 3149 heartbeater.cc:511] Master 127.13.99.62:36325 requested a full tablet report, sending...
I20260812 09:47:50.846884 2262 ts_manager.cc:194] Registered new tserver with Master: a6bf7dc8e82242c3950f456c5de6a03c (127.13.99.3:46447)
I20260812 09:47:50.848227 2262 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.99.3@KRBTEST.COM'} at 127.13.99.3:35719
Aug 12 09:47:51 dist-test-slave-jq4t krb5kdc[475](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786528071, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:47:51.259817 2627 heartbeater.cc:464] Registering TS with master...
I20260812 09:47:51.260128 2627 heartbeater.cc:511] Master 127.13.99.62:36325 requested a full tablet report, sending...
I20260812 09:47:51.260903 2262 ts_manager.cc:194] Registered new tserver with Master: bd25d8d5808244d286f3dd6062abb17e (127.13.99.1:33427)
I20260812 09:47:51.261716 2262 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.99.1@KRBTEST.COM'} at 127.13.99.1:35841
Aug 12 09:47:51 dist-test-slave-jq4t krb5kdc[475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528071, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.62@KRBTEST.COM for HTTP/127.13.99.20@KRBTEST.COM
I20260812 09:47:51.573277 2882 heartbeater.cc:464] Registering TS with master...
I20260812 09:47:51.573604 2882 heartbeater.cc:511] Master 127.13.99.62:36325 requested a full tablet report, sending...
I20260812 09:47:51.574342 2262 ts_manager.cc:194] Registered new tserver with Master: 05f1de16c6dd4cbcaf2e3a33fd060f16 (127.13.99.2:45277)
I20260812 09:47:51.575127 2262 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.99.2@KRBTEST.COM'} at 127.13.99.2:44773
I20260812 09:47:51.584322 13708 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 09:47:51.603470 3177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:47:51.588978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36586 (local address 127.13.99.62:36325)
0812 09:47:51.590013 (+ 1035us) server_negotiation.cc:206] Beginning negotiation
0812 09:47:51.590018 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:47:51.590032 (+ 14us) server_negotiation.cc:407] Connection header received
0812 09:47:51.590077 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:47:51.590080 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:47:51.590130 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:47:51.591236 (+ 1106us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:47:51.591345 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:47:51.592332 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:47:51.593615 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:47:51.593965 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:47:51.594643 (+ 678us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:47:51.599027 (+ 4384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:47:51.599032 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:47:51.599034 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:47:51.599065 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:47:51.601021 (+ 1956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:47:51.601563 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:47:51.601567 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:47:51.601569 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:47:51.601660 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:47:51.601968 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:47:51.601971 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:47:51.601973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:47:51.602207 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:47:51.603063 (+ 856us) server_negotiation.cc:1035] Waiting for connection context
0812 09:47:51.603142 (+ 79us) server_negotiation.cc:299] Negotiation successful
0812 09:47:51.603272 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":940,"thread_start_us":113,"threads_started":1}
I20260812 09:47:51.607784 2262 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36586:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 09:47:51.612633 2262 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 09:47:51.641479 3192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:47:51.625940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.1:33427 (local address 127.0.0.1:44444)
0812 09:47:51.632807 (+ 6867us) negotiation.cc:107] Waiting for socket to connect
0812 09:47:51.632881 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0812 09:47:51.633031 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:47:51.635909 (+ 2878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:47:51.635912 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:47:51.635944 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:47:51.636033 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:47:51.636039 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:47:51.637570 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:47:51.637574 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:47:51.639602 (+ 2028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:47:51.639615 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:47:51.640902 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:47:51.640905 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:47:51.641030 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:47:51.641066 (+ 36us) client_negotiation.cc:769] Sending connection context
0812 09:47:51.641182 (+ 116us) client_negotiation.cc:240] Negotiation successful
0812 09:47:51.641311 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6694,"thread_start_us":80,"threads_started":1}
I20260812 09:47:51.642130 3193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:47:51.626613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44444 (local address 127.13.99.1:33427)
0812 09:47:51.635302 (+ 8689us) server_negotiation.cc:206] Beginning negotiation
0812 09:47:51.635309 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 09:47:51.635333 (+ 24us) server_negotiation.cc:407] Connection header received
0812 09:47:51.635408 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:47:51.635414 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:47:51.635593 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:47:51.635731 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:47:51.636170 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:47:51.637427 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:47:51.639741 (+ 2314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:47:51.640584 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:47:51.640690 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:47:51.640840 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0812 09:47:51.641888 (+ 1048us) server_negotiation.cc:299] Negotiation successful
0812 09:47:51.641995 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8569,"thread_start_us":97,"threads_started":1}
I20260812 09:47:51.642755 3187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:47:51.624444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.3:46447 (local address 127.0.0.1:51106)
0812 09:47:51.632808 (+ 8364us) negotiation.cc:107] Waiting for socket to connect
0812 09:47:51.632877 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0812 09:47:51.633025 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:47:51.634161 (+ 1136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:47:51.634168 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:47:51.634196 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:47:51.634353 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:47:51.634371 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:47:51.639925 (+ 5554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:47:51.639929 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:47:51.641190 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:47:51.641201 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:47:51.642370 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:47:51.642375 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:47:51.642480 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:47:51.642505 (+ 25us) client_negotiation.cc:769] Sending connection context
0812 09:47:51.642556 (+ 51us) client_negotiation.cc:240] Negotiation successful
0812 09:47:51.642622 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8136,"thread_start_us":111,"threads_started":1}
I20260812 09:47:51.643491 3188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:47:51.624641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51106 (local address 127.13.99.3:46447)
0812 09:47:51.633612 (+ 8971us) server_negotiation.cc:206] Beginning negotiation
0812 09:47:51.633620 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 09:47:51.633647 (+ 27us) server_negotiation.cc:407] Connection header received
0812 09:47:51.633731 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:47:51.633737 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:47:51.633994 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:47:51.634173 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:47:51.636883 (+ 2710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:47:51.638051 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:47:51.641380 (+ 3329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:47:51.642206 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:47:51.643053 (+ 847us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:47:51.643200 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0812 09:47:51.643251 (+ 51us) server_negotiation.cc:299] Negotiation successful
0812 09:47:51.643338 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8849,"thread_start_us":140,"threads_started":1}
I20260812 09:47:51.643893 3190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:47:51.626222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.2:45277 (local address 127.0.0.1:57156)
0812 09:47:51.633254 (+ 7032us) negotiation.cc:107] Waiting for socket to connect
0812 09:47:51.633267 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 09:47:51.633340 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:47:51.635259 (+ 1919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:47:51.635263 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:47:51.635281 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:47:51.635381 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:47:51.635386 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:47:51.641231 (+ 5845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:47:51.641234 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:47:51.642288 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:47:51.642302 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:47:51.643530 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:47:51.643534 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:47:51.643617 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:47:51.643639 (+ 22us) client_negotiation.cc:769] Sending connection context
0812 09:47:51.643710 (+ 71us) client_negotiation.cc:240] Negotiation successful
0812 09:47:51.643774 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6974,"thread_start_us":80,"threads_started":1}
I20260812 09:47:51.644302 3195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:47:51.626966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57156 (local address 127.13.99.2:45277)
0812 09:47:51.634655 (+ 7689us) server_negotiation.cc:206] Beginning negotiation
0812 09:47:51.634662 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 09:47:51.634686 (+ 24us) server_negotiation.cc:407] Connection header received
0812 09:47:51.634764 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:47:51.634769 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:47:51.634952 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:47:51.635089 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:47:51.639982 (+ 4893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:47:51.641099 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:47:51.642462 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:47:51.643216 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:47:51.643323 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:47:51.643464 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0812 09:47:51.644065 (+ 601us) server_negotiation.cc:299] Negotiation successful
0812 09:47:51.644164 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7572,"thread_start_us":88,"threads_started":1}
I20260812 09:47:51.645569 2505 tablet_service.cc:1467] Processing CreateTablet for tablet 7891e8459b094e15a15bdf51ffed01db (DEFAULT_TABLE table=test-table [id=eef9be99b5eb412d8e3be051b38c8f96]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 09:47:51.646284 3015 tablet_service.cc:1467] Processing CreateTablet for tablet 7891e8459b094e15a15bdf51ffed01db (DEFAULT_TABLE table=test-table [id=eef9be99b5eb412d8e3be051b38c8f96]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 09:47:51.646783 2505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7891e8459b094e15a15bdf51ffed01db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:47:51.647352 3015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7891e8459b094e15a15bdf51ffed01db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:47:51.647403 2728 tablet_service.cc:1467] Processing CreateTablet for tablet 7891e8459b094e15a15bdf51ffed01db (DEFAULT_TABLE table=test-table [id=eef9be99b5eb412d8e3be051b38c8f96]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 09:47:51.648360 2728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7891e8459b094e15a15bdf51ffed01db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:47:51.656697 3202 tablet_bootstrap.cc:492] T 7891e8459b094e15a15bdf51ffed01db P 05f1de16c6dd4cbcaf2e3a33fd060f16: Bootstrap starting.
I20260812 09:47:51.656940 3198 tablet_bootstrap.cc:492] T 7891e8459b094e15a15bdf51ffed01db P a6bf7dc8e82242c3950f456c5de6a03c: Bootstrap starting.
I20260812 09:47:51.659021 3202 tablet_bootstrap.cc:654] T 7891e8459b094e15a15bdf51ffed01db P 05f1de16c6dd4cbcaf2e3a33fd060f16: Neither blocks nor log segments found. Creating new log.
I20260812 09:47:51.659901 3202 log.cc:824] T 7891e8459b094e15a15bdf51ffed01db P 05f1de16c6dd4cbcaf2e3a33fd060f16: Log is configured to *not* fsync() on all Append() calls
I20260812 09:47:51.660247 3200 tablet_bootstrap.cc:492] T 7891e8459b094e15a15bdf51ffed01db P bd25d8d5808244d286f3dd6062abb17e: Bootstrap starting.
I20260812 09:47:51.662073 3202 tablet_bootstrap.cc:492] T 7891e8459b094e15a15bdf51ffed01db P 05f1de16c6dd4cbcaf2e3a33fd060f16: No bootstrap required, opened a new log
I20260812 09:47:51.662279 3202 ts_tablet_manager.cc:1397] T 7891e8459b094e15a15bdf51ffed01db P 05f1de16c6dd4cbcaf2e3a33fd060f16: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260812 09:47:51.662379 3200 tablet_bootstrap.cc:654] T 7891e8459b094e15a15bdf51ffed01db P bd25d8d5808244d286f3dd6062abb17e: Neither blocks nor log segments found. Creating new log.
I20260812 09:47:51.663232 3200 log.cc:824] T 7891e8459b094e15a15bdf51ffed01db P bd25d8d5808244d286f3dd6062abb17e: Log is configured to *not* fsync() on all Append() calls
I20260812 09:47:51.664498 3198 tablet_bootstrap.cc:654] T 7891e8459b094e15a15bdf51ffed01db P a6bf7dc8e82242c3950f456c5de6a03c: Neither blocks nor log segments found. Creating new log.
I20260812 09:47:51.665298 3198 log.cc:824] T 7891e8459b094e15a15bdf51ffed01db P a6bf7dc8e82242c3950f456c5de6a03c: Log is configured to *not* fsync() on all Append() calls
I20260812 09:47:51.666507 3202 raft_consensus.cc:348] T 7891e8459b094e15a15bdf51ffed01db P 05f1de16c6dd4cbcaf2e3a33fd060f16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6bf7dc8e82242c3950f456c5de6a03c" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 46447 } } peers { permanent_uuid: "bd25d8d5808244d286f3dd6062abb17e" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 33427 } } peers { permanent_uuid: "05f1de16c6dd4cbcaf2e3a33fd060f16" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 45277 } }
I20260812 09:47:51.666779 3202 raft_consensus.cc:374] T 7891e8459b094e15a15bdf51ffed01db P 05f1de16c6dd4cbcaf2e3a33fd060f16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:47:51.666836 3202 raft_consensus.cc:729] T 7891e8459b094e15a15bdf51ffed01db P 05f1de16c6dd4cbcaf2e3a33fd060f16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f1de16c6dd4cbcaf2e3a33fd060f16, State: Initialized, Role: FOLLOWER
I20260812 09:47:51.667374 3202 consensus_queue.cc:260] T 7891e8459b094e15a15bdf51ffed01db P 05f1de16c6dd4cbcaf2e3a33fd060f16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6bf7dc8e82242c3950f456c5de6a03c" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 46447 } } peers { permanent_uuid: "bd25d8d5808244d286f3dd6062abb17e" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 33427 } } peers { permanent_uuid: "05f1de16c6dd4cbcaf2e3a33fd060f16" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 45277 } }
I20260812 09:47:51.668267 3200 tablet_bootstrap.cc:492] T 7891e8459b094e15a15bdf51ffed01db P bd25d8d5808244d286f3dd6062abb17e: No bootstrap required, opened a new log
I20260812 09:47:51.668398 3202 ts_tablet_manager.cc:1428] T 7891e8459b094e15a15bdf51ffed01db P 05f1de16c6dd4cbcaf2e3a33fd060f16: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260812 09:47:51.668498 3200 ts_tablet_manager.cc:1397] T 7891e8459b094e15a15bdf51ffed01db P bd25d8d5808244d286f3dd6062abb17e: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260812 09:47:51.668797 2882 heartbeater.cc:503] Master 127.13.99.62:36325 was elected leader, sending a full tablet report...
I20260812 09:47:51.672390 3200 raft_consensus.cc:348] T 7891e8459b094e15a15bdf51ffed01db P bd25d8d5808244d286f3dd6062abb17e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6bf7dc8e82242c3950f456c5de6a03c" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 46447 } } peers { permanent_uuid: "bd25d8d5808244d286f3dd6062abb17e" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 33427 } } peers { permanent_uuid: "05f1de16c6dd4cbcaf2e3a33fd060f16" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 45277 } }
I20260812 09:47:51.673017 3200 raft_consensus.cc:374] T 7891e8459b094e15a15bdf51ffed01db P bd25d8d5808244d286f3dd6062abb17e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:47:51.673084 3200 raft_consensus.cc:729] T 7891e8459b094e15a15bdf51ffed01db P bd25d8d5808244d286f3dd6062abb17e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd25d8d5808244d286f3dd6062abb17e, State: Initialized, Role: FOLLOWER
I20260812 09:47:51.673534 3200 consensus_queue.cc:260] T 7891e8459b094e15a15bdf51ffed01db P bd25d8d5808244d286f3dd6062abb17e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6bf7dc8e82242c3950f456c5de6a03c" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 46447 } } peers { permanent_uuid: "bd25d8d5808244d286f3dd6062abb17e" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 33427 } } peers { permanent_uuid: "05f1de16c6dd4cbcaf2e3a33fd060f16" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 45277 } }
I20260812 09:47:51.674594 3200 ts_tablet_manager.cc:1428] T 7891e8459b094e15a15bdf51ffed01db P bd25d8d5808244d286f3dd6062abb17e: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260812 09:47:51.674701 2627 heartbeater.cc:503] Master 127.13.99.62:36325 was elected leader, sending a full tablet report...
I20260812 09:47:51.676265 3198 tablet_bootstrap.cc:492] T 7891e8459b094e15a15bdf51ffed01db P a6bf7dc8e82242c3950f456c5de6a03c: No bootstrap required, opened a new log
I20260812 09:47:51.676504 3198 ts_tablet_manager.cc:1397] T 7891e8459b094e15a15bdf51ffed01db P a6bf7dc8e82242c3950f456c5de6a03c: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.000s
I20260812 09:47:51.681277 3198 raft_consensus.cc:348] T 7891e8459b094e15a15bdf51ffed01db P a6bf7dc8e82242c3950f456c5de6a03c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6bf7dc8e82242c3950f456c5de6a03c" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 46447 } } peers { permanent_uuid: "bd25d8d5808244d286f3dd6062abb17e" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 33427 } } peers { permanent_uuid: "05f1de16c6dd4cbcaf2e3a33fd060f16" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 45277 } }
I20260812 09:47:51.681571 3198 raft_consensus.cc:374] T 7891e8459b094e15a15bdf51ffed01db P a6bf7dc8e82242c3950f456c5de6a03c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:47:51.681640 3198 raft_consensus.cc:729] T 7891e8459b094e15a15bdf51ffed01db P a6bf7dc8e82242c3950f456c5de6a03c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6bf7dc8e82242c3950f456c5de6a03c, State: Initialized, Role: FOLLOWER
I20260812 09:47:51.682106 3198 consensus_queue.cc:260] T 7891e8459b094e15a15bdf51ffed01db P a6bf7dc8e82242c3950f456c5de6a03c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6bf7dc8e82242c3950f456c5de6a03c" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 46447 } } peers { permanent_uuid: "bd25d8d5808244d286f3dd6062abb17e" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 33427 } } peers { permanent_uuid: "05f1de16c6dd4cbcaf2e3a33fd060f16" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 45277 } }
I20260812 09:47:51.682797 3149 heartbeater.cc:503] Master 127.13.99.62:36325 was elected leader, sending a full tablet report...
I20260812 09:47:51.683168 3198 ts_tablet_manager.cc:1428] T 7891e8459b094e15a15bdf51ffed01db P a6bf7dc8e82242c3950f456c5de6a03c: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260812 09:47:51.683604 3219 raft_consensus.cc:482] T 7891e8459b094e15a15bdf51ffed01db P a6bf7dc8e82242c3950f456c5de6a03c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 09:47:51.683758 3219 raft_consensus.cc:504] T 7891e8459b094e15a15bdf51ffed01db P a6bf7dc8e82242c3950f456c5de6a03c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6bf7dc8e82242c3950f456c5de6a03c" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 46447 } } peers { permanent_uuid: "bd25d8d5808244d286f3dd6062abb17e" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 33427 } } peers { permanent_uuid: "05f1de16c6dd4cbcaf2e3a33fd060f16" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 45277 } }
I20260812 09:47:51.685047 3219 leader_election.cc:288] T 7891e8459b094e15a15bdf51ffed01db P a6bf7dc8e82242c3950f456c5de6a03c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd25d8d5808244d286f3dd6062abb17e (127.13.99.1:33427), 05f1de16c6dd4cbcaf2e3a33fd060f16 (127.13.99.2:45277)
I20260812 09:47:51.690791 3193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:47:51.686216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.3:43019 (local address 127.13.99.1:33427)
0812 09:47:51.686340 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0812 09:47:51.686350 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 09:47:51.686364 (+ 14us) server_negotiation.cc:407] Connection header received
0812 09:47:51.686412 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:47:51.686415 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:47:51.686468 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:47:51.686548 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:47:51.687033 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:47:51.687850 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:47:51.689483 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:47:51.690082 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:47:51.690160 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:47:51.690224 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0812 09:47:51.690573 (+ 349us) server_negotiation.cc:299] Negotiation successful
0812 09:47:51.690667 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260812 09:47:51.691264 3224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:47:51.685467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.1:33427 (local address 127.13.99.3:43019)
0812 09:47:51.686129 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0812 09:47:51.686145 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 09:47:51.686263 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:47:51.686709 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:47:51.686712 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:47:51.686732 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:47:51.686880 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:47:51.686887 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:47:51.687996 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:47:51.687999 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:47:51.689326 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:47:51.689338 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:47:51.690318 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:47:51.690322 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:47:51.690435 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:47:51.690459 (+ 24us) client_negotiation.cc:769] Sending connection context
0812 09:47:51.691031 (+ 572us) client_negotiation.cc:240] Negotiation successful
0812 09:47:51.691108 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":91,"threads_started":1}
I20260812 09:47:51.691850 3195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:47:51.685916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.3:55575 (local address 127.13.99.2:45277)
0812 09:47:51.686054 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0812 09:47:51.686059 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:47:51.686921 (+ 862us) server_negotiation.cc:407] Connection header received
0812 09:47:51.687173 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:47:51.687177 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:47:51.687225 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:47:51.687280 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:47:51.687710 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:47:51.688540 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:47:51.690064 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:47:51.690655 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:47:51.690735 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:47:51.690827 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0812 09:47:51.691643 (+ 816us) server_negotiation.cc:299] Negotiation successful
0812 09:47:51.691732 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260812 09:47:51.691879 2531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7891e8459b094e15a15bdf51ffed01db" candidate_uuid: "a6bf7dc8e82242c3950f456c5de6a03c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd25d8d5808244d286f3dd6062abb17e" is_pre_election: true
I20260812 09:47:51.692226 3222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:47:51.685815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.2:45277 (local address 127.13.99.3:55575)
0812 09:47:51.686822 (+ 1007us) negotiation.cc:107] Waiting for socket to connect
0812 09:47:51.686843 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 09:47:51.687041 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:47:51.687421 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:47:51.687424 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:47:51.687444 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:47:51.687565 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:47:51.687571 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:47:51.688687 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:47:51.688690 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:47:51.689884 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:47:51.689897 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:47:51.691171 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:47:51.691175 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:47:51.691279 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:47:51.691300 (+ 21us) client_negotiation.cc:769] Sending connection context
0812 09:47:51.692006 (+ 706us) client_negotiation.cc:240] Negotiation successful
0812 09:47:51.692082 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":943,"thread_start_us":92,"threads_started":1}
I20260812 09:47:51.692211 2531 raft_consensus.cc:2454] T 7891e8459b094e15a15bdf51ffed01db P bd25d8d5808244d286f3dd6062abb17e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6bf7dc8e82242c3950f456c5de6a03c in term 0.
I20260812 09:47:51.692761 2917 leader_election.cc:302] T 7891e8459b094e15a15bdf51ffed01db P a6bf7dc8e82242c3950f456c5de6a03c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6bf7dc8e82242c3950f456c5de6a03c, bd25d8d5808244d286f3dd6062abb17e; no voters:
I20260812 09:47:51.692914 2795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7891e8459b094e15a15bdf51ffed01db" candidate_uuid: "a6bf7dc8e82242c3950f456c5de6a03c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f1de16c6dd4cbcaf2e3a33fd060f16" is_pre_election: true
I20260812 09:47:51.693126 3220 raft_consensus.cc:2781] T 7891e8459b094e15a15bdf51ffed01db P a6bf7dc8e82242c3950f456c5de6a03c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 09:47:51.693182 2795 raft_consensus.cc:2454] T 7891e8459b094e15a15bdf51ffed01db P 05f1de16c6dd4cbcaf2e3a33fd060f16 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6bf7dc8e82242c3950f456c5de6a03c in term 0.
I20260812 09:47:51.693238 3220 raft_consensus.cc:482] T 7891e8459b094e15a15bdf51ffed01db P a6bf7dc8e82242c3950f456c5de6a03c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 09:47:51.693298 3220 raft_consensus.cc:3037] T 7891e8459b094e15a15bdf51ffed01db P a6bf7dc8e82242c3950f456c5de6a03c [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:47:51.694676 3220 raft_consensus.cc:504] T 7891e8459b094e15a15bdf51ffed01db P a6bf7dc8e82242c3950f456c5de6a03c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6bf7dc8e82242c3950f456c5de6a03c" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 46447 } } peers { permanent_uuid: "bd25d8d5808244d286f3dd6062abb17e" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 33427 } } peers { permanent_uuid: "05f1de16c6dd4cbcaf2e3a33fd060f16" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 45277 } }
I20260812 09:47:51.695046 3220 leader_election.cc:288] T 7891e8459b094e15a15bdf51ffed01db P a6bf7dc8e82242c3950f456c5de6a03c [CANDIDATE]: Term 1 election: Requested vote from peers bd25d8d5808244d286f3dd6062abb17e (127.13.99.1:33427), 05f1de16c6dd4cbcaf2e3a33fd060f16 (127.13.99.2:45277)
I20260812 09:47:51.695365 2531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7891e8459b094e15a15bdf51ffed01db" candidate_uuid: "a6bf7dc8e82242c3950f456c5de6a03c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd25d8d5808244d286f3dd6062abb17e"
I20260812 09:47:51.695458 2795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7891e8459b094e15a15bdf51ffed01db" candidate_uuid: "a6bf7dc8e82242c3950f456c5de6a03c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f1de16c6dd4cbcaf2e3a33fd060f16"
I20260812 09:47:51.695503 2531 raft_consensus.cc:3037] T 7891e8459b094e15a15bdf51ffed01db P bd25d8d5808244d286f3dd6062abb17e [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:47:51.695578 2795 raft_consensus.cc:3037] T 7891e8459b094e15a15bdf51ffed01db P 05f1de16c6dd4cbcaf2e3a33fd060f16 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:47:51.696836 2795 raft_consensus.cc:2454] T 7891e8459b094e15a15bdf51ffed01db P 05f1de16c6dd4cbcaf2e3a33fd060f16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6bf7dc8e82242c3950f456c5de6a03c in term 1.
I20260812 09:47:51.696836 2531 raft_consensus.cc:2454] T 7891e8459b094e15a15bdf51ffed01db P bd25d8d5808244d286f3dd6062abb17e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6bf7dc8e82242c3950f456c5de6a03c in term 1.
I20260812 09:47:51.697115 2917 leader_election.cc:302] T 7891e8459b094e15a15bdf51ffed01db P a6bf7dc8e82242c3950f456c5de6a03c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f1de16c6dd4cbcaf2e3a33fd060f16, a6bf7dc8e82242c3950f456c5de6a03c; no voters:
I20260812 09:47:51.697301 3220 raft_consensus.cc:2781] T 7891e8459b094e15a15bdf51ffed01db P a6bf7dc8e82242c3950f456c5de6a03c [term 1 FOLLOWER]: Leader election won for term 1
I20260812 09:47:51.697674 3220 raft_consensus.cc:686] T 7891e8459b094e15a15bdf51ffed01db P a6bf7dc8e82242c3950f456c5de6a03c [term 1 LEADER]: Becoming Leader. State: Replica: a6bf7dc8e82242c3950f456c5de6a03c, State: Running, Role: LEADER
I20260812 09:47:51.698141 3220 consensus_queue.cc:237] T 7891e8459b094e15a15bdf51ffed01db P a6bf7dc8e82242c3950f456c5de6a03c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6bf7dc8e82242c3950f456c5de6a03c" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 46447 } } peers { permanent_uuid: "bd25d8d5808244d286f3dd6062abb17e" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 33427 } } peers { permanent_uuid: "05f1de16c6dd4cbcaf2e3a33fd060f16" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 45277 } }
I20260812 09:47:51.701344 2262 catalog_manager.cc:5495] T 7891e8459b094e15a15bdf51ffed01db P a6bf7dc8e82242c3950f456c5de6a03c reported cstate change: term changed from 0 to 1, leader changed from <none> to a6bf7dc8e82242c3950f456c5de6a03c (127.13.99.3). New cstate: current_term: 1 leader_uuid: "a6bf7dc8e82242c3950f456c5de6a03c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6bf7dc8e82242c3950f456c5de6a03c" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 46447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bd25d8d5808244d286f3dd6062abb17e" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 33427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05f1de16c6dd4cbcaf2e3a33fd060f16" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 45277 } health_report { overall_health: UNKNOWN } } }
W20260812 09:47:51.722317 2628 tablet.cc:2369] T 7891e8459b094e15a15bdf51ffed01db P bd25d8d5808244d286f3dd6062abb17e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 09:47:51.723162 3188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:47:51.718260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51118 (local address 127.13.99.3:46447)
0812 09:47:51.718421 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0812 09:47:51.718426 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:47:51.718455 (+ 29us) server_negotiation.cc:407] Connection header received
0812 09:47:51.718642 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:47:51.718644 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:47:51.718702 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:47:51.718829 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 09:47:51.719116 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:47:51.719730 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:47:51.720806 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:47:51.721038 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:47:51.721154 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:47:51.721478 (+ 324us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 09:47:51.721924 (+ 446us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 09:47:51.722068 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0812 09:47:51.723001 (+ 933us) server_negotiation.cc:299] Negotiation successful
0812 09:47:51.723057 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260812 09:47:51.729249 2795 raft_consensus.cc:1261] T 7891e8459b094e15a15bdf51ffed01db P 05f1de16c6dd4cbcaf2e3a33fd060f16 [term 1 FOLLOWER]: Refusing update from remote peer a6bf7dc8e82242c3950f456c5de6a03c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 09:47:51.729249 2531 raft_consensus.cc:1261] T 7891e8459b094e15a15bdf51ffed01db P bd25d8d5808244d286f3dd6062abb17e [term 1 FOLLOWER]: Refusing update from remote peer a6bf7dc8e82242c3950f456c5de6a03c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 09:47:51.730029 3220 consensus_queue.cc:1035] T 7891e8459b094e15a15bdf51ffed01db P a6bf7dc8e82242c3950f456c5de6a03c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd25d8d5808244d286f3dd6062abb17e" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 33427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 09:47:51.730258 3219 consensus_queue.cc:1035] T 7891e8459b094e15a15bdf51ffed01db P a6bf7dc8e82242c3950f456c5de6a03c [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f1de16c6dd4cbcaf2e3a33fd060f16" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 45277 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 09:47:51.740605 3228 mvcc.cc:205] Tried to move back new op lower bound from 7317618981795094528 to 7317618981678608384. Current Snapshot: MvccSnapshot[applied={T|T < 7317618981795094528}]
I20260812 09:47:51.743568 3234 mvcc.cc:205] Tried to move back new op lower bound from 7317618981795094528 to 7317618981678608384. Current Snapshot: MvccSnapshot[applied={T|T < 7317618981795094528}]
I20260812 09:47:51.750550 3238 mvcc.cc:205] Tried to move back new op lower bound from 7317618981795094528 to 7317618981678608384. Current Snapshot: MvccSnapshot[applied={T|T < 7317618981795094528}]
I20260812 09:47:51.759729 2261 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36586:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 09:47:51.759928 2261 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36586:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 09:47:51.762744 2261 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a504c7abdfd44aaebb24958ae0f857e4: Sending DeleteTablet for 3 replicas of tablet 7891e8459b094e15a15bdf51ffed01db
I20260812 09:47:51.764020 3015 tablet_service.cc:1514] Processing DeleteTablet for tablet 7891e8459b094e15a15bdf51ffed01db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 09:47:51 UTC) from {username='kudu', principal='kudu/127.13.99.62@KRBTEST.COM'} at 127.0.0.1:51106
I20260812 09:47:51.764016 2728 tablet_service.cc:1514] Processing DeleteTablet for tablet 7891e8459b094e15a15bdf51ffed01db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 09:47:51 UTC) from {username='kudu', principal='kudu/127.13.99.62@KRBTEST.COM'} at 127.0.0.1:57156
I20260812 09:47:51.764737 2505 tablet_service.cc:1514] Processing DeleteTablet for tablet 7891e8459b094e15a15bdf51ffed01db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 09:47:51 UTC) from {username='kudu', principal='kudu/127.13.99.62@KRBTEST.COM'} at 127.0.0.1:44444
I20260812 09:47:51.765944 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 2341
I20260812 09:47:51.768682 3254 tablet_replica.cc:330] T 7891e8459b094e15a15bdf51ffed01db P a6bf7dc8e82242c3950f456c5de6a03c: stopping tablet replica
I20260812 09:47:51.769107 3254 raft_consensus.cc:2229] T 7891e8459b094e15a15bdf51ffed01db P a6bf7dc8e82242c3950f456c5de6a03c [term 1 LEADER]: Raft consensus shutting down.
I20260812 09:47:51.769554 3254 raft_consensus.cc:2258] T 7891e8459b094e15a15bdf51ffed01db P a6bf7dc8e82242c3950f456c5de6a03c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 09:47:51.779063 3254 ts_tablet_manager.cc:1905] T 7891e8459b094e15a15bdf51ffed01db P a6bf7dc8e82242c3950f456c5de6a03c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 09:47:51.782855 3254 ts_tablet_manager.cc:1918] T 7891e8459b094e15a15bdf51ffed01db P a6bf7dc8e82242c3950f456c5de6a03c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 09:47:51.783252 3250 tablet_replica.cc:330] T 7891e8459b094e15a15bdf51ffed01db P 05f1de16c6dd4cbcaf2e3a33fd060f16: stopping tablet replica
I20260812 09:47:51.783444 3254 log.cc:1196] T 7891e8459b094e15a15bdf51ffed01db P a6bf7dc8e82242c3950f456c5de6a03c: Deleting WAL directory at /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786527905902985-13708-0/minicluster-data/ts-2/wal/wals/7891e8459b094e15a15bdf51ffed01db
I20260812 09:47:51.783898 3250 raft_consensus.cc:2229] T 7891e8459b094e15a15bdf51ffed01db P 05f1de16c6dd4cbcaf2e3a33fd060f16 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 09:47:51.784421 3254 ts_tablet_manager.cc:1939] T 7891e8459b094e15a15bdf51ffed01db P a6bf7dc8e82242c3950f456c5de6a03c: Deleting consensus metadata
I20260812 09:47:51.784902 3250 raft_consensus.cc:2258] T 7891e8459b094e15a15bdf51ffed01db P 05f1de16c6dd4cbcaf2e3a33fd060f16 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 09:47:51.786583 2227 catalog_manager.cc:4841] TS a6bf7dc8e82242c3950f456c5de6a03c (127.13.99.3:46447): tablet 7891e8459b094e15a15bdf51ffed01db (table test-table [id=eef9be99b5eb412d8e3be051b38c8f96]) successfully deleted
W20260812 09:47:51.787111 2227 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.99.1:33427 (error 108)
I20260812 09:47:51.788905 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 2632
I20260812 09:47:51.788939 3190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:47:51.788622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.1:33427 (local address 127.0.0.1:44450)
0812 09:47:51.788771 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0812 09:47:51.788847 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.1:33427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260812 09:47:51.793175 2227 catalog_manager.cc:4584] TS bd25d8d5808244d286f3dd6062abb17e (127.13.99.1:33427): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7891e8459b094e15a15bdf51ffed01db: Network error: Client connection negotiation failed: client connection to 127.13.99.1:33427: connect: Connection refused (error 111)
I20260812 09:47:51.847517 3190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:47:51.847146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.1:33427 (local address 127.0.0.1:44462)
0812 09:47:51.847356 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0812 09:47:51.847417 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.1:33427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
I20260812 09:47:51.848500 3190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:47:51.848217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.1:33427 (local address 127.0.0.1:44466)
0812 09:47:51.848355 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0812 09:47:51.848408 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.1:33427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260812 09:47:51.855901 3190 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 09:47:51.855574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.2:45277 (local address 127.0.0.1:57166)
0812 09:47:51.855705 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0812 09:47:51.855722 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 09:47:51.855806 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.2:45277: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":39}
I20260812 09:47:51.856196 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 2885
I20260812 09:47:51.870967 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 2194
I20260812 09:47:51.891067 13708 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260812 09:47:52.236303 13708 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260812 09:47:52.236449 13708 mini_ranger.cc:66] Stopping Ranger...
I20260812 09:47:52.581362 13708 mini_ranger.cc:68] Stopped Ranger
2026-08-12 09:47:52.581 UTC [530] LOG: received smart shutdown request
2026-08-12 09:47:52.586 UTC [530] LOG: background worker "logical replication launcher" (PID 546) exited with exit code 1
2026-08-12 09:47:52.586 UTC [535] LOG: shutting down
2026-08-12 09:47:52.764 UTC [530] LOG: database system is shut down
2026-08-12T09:47:52Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (49286 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786527905902985-13708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 09:47:52 dist-test-slave-jq4t krb5kdc[3278](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 09:47:52 dist-test-slave-jq4t krb5kdc[3278](info): set up 2 sockets
Aug 12 09:47:52 dist-test-slave-jq4t krb5kdc[3278](info): commencing operation
krb5kdc: starting...
W20260812 09:47:54.210510 13708 mini_kdc.cc:121] Time spent starting KDC: real 1.293s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786527905902985-13708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786527905902985-13708-0/krb5kdc/test-user.keytab.
Aug 12 09:47:54 dist-test-slave-jq4t krb5kdc[3278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528074, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T09:47:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T09:47:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.99.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.99.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.99.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab.
Entry for principal kudu/127.13.99.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab.
I20260812 09:47:54.343523 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786527905902985-13708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786527905902985-13708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786527905902985-13708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.62:42443
--webserver_interface=127.13.99.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab
--principal=kudu/127.13.99.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:40175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.99.62:42443,127.13.99.61:43587,127.13.99.60:38807
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:47:54.462586 3298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:47:54.462901 3298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:47:54.462990 3298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:47:54.467163 3298 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 09:47:54.467254 3298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:47:54.467293 3298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:47:54.467329 3298 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 09:47:54.467365 3298 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 09:47:54.472467 3298 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:40175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786527905902985-13708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786527905902985-13708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.99.62:42443,127.13.99.61:43587,127.13.99.60:38807
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab
--principal=kudu/127.13.99.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.62:42443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.99.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3298
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786527905902985-13708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:47:54.473538 3298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:47:54.474551 3298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:47:54.483156 3309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:47:54.483247 3313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:47:54.483237 3298 server_base.cc:1034] running on GCE node
W20260812 09:47:54.483330 3307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:47:54.483943 3298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:47:54.485105 3298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:47:54.486312 3298 hybrid_clock.cc:648] HybridClock initialized: now 1786528074486293 us; error 25 us; skew 500 ppm
Aug 12 09:47:54 dist-test-slave-jq4t krb5kdc[3278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528074, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:47:54.489264 3298 init.cc:377] Logged in from keytab as kudu/127.13.99.62@KRBTEST.COM (short username kudu)
I20260812 09:47:54.490222 3298 webserver.cc:460] Webserver started at http://127.13.99.62:35265/ using document root <none> and password file <none>
I20260812 09:47:54.490783 3298 fs_manager.cc:359] Metadata directory not provided
I20260812 09:47:54.490857 3298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:47:54.491112 3298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:47:54.493244 3298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786527905902985-13708-0/minicluster-data/master-0/data/instance:
uuid: "4f61b4a745f74e73ae77dfa285a9d849"
format_stamp: "Formatted at 2026-08-12 09:47:54 on dist-test-slave-jq4t"
server_key: "8bf97e11fa3c471572d54e9fea993896"
server_key_iv: "c1f0f1a43d9a7db382909342e40ff787"
server_key_version: "encryptionkey@0"
I20260812 09:47:54.493829 3298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786527905902985-13708-0/minicluster-data/master-0/wal/instance:
uuid: "4f61b4a745f74e73ae77dfa285a9d849"
format_stamp: "Formatted at 2026-08-12 09:47:54 on dist-test-slave-jq4t"
server_key: "8bf97e11fa3c471572d54e9fea993896"
server_key_iv: "c1f0f1a43d9a7db382909342e40ff787"
server_key_version: "encryptionkey@0"
I20260812 09:47:54.497622 3298 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 09:47:54.500003 3323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:47:54.501281 3298 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 09:47:54.501435 3298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786527905902985-13708-0/minicluster-data/master-0/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786527905902985-13708-0/minicluster-data/master-0/wal
uuid: "4f61b4a745f74e73ae77dfa285a9d849"
format_stamp: "Formatted at 2026-08-12 09:47:54 on dist-test-slave-jq4t"
server_key: "8bf97e11fa3c471572d54e9fea993896"
server_key_iv: "c1f0f1a43d9a7db382909342e40ff787"
server_key_version: "encryptionkey@0"
I20260812 09:47:54.501551 3298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786527905902985-13708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786527905902985-13708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786527905902985-13708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:47:54.520094 3298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:47:54.523262 3298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:47:54.523489 3298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:47:54.531282 3298 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.62:42443
I20260812 09:47:54.531292 3416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.62:42443 every 8 connection(s)
I20260812 09:47:54.532294 3298 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
I20260812 09:47:54.535202 3425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:47:54.539371 3425 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.99.62" port: 42443 } has no permanent_uuid. Determining permanent_uuid...
I20260812 09:47:54.540740 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 3298
I20260812 09:47:54.540884 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786527905902985-13708-0/minicluster-data/master-0/wal/instance
I20260812 09:47:54.541252 13708 external_mini_cluster.cc:1428] Setting key a1d3543bd0166d3f58ff64b5c0b312bc
Aug 12 09:47:54 dist-test-slave-jq4t krb5kdc[3278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528074, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.62@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.99.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.99.61@KRBTEST.COM" created.
I20260812 09:47:54.556368 3429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:47:54.541834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:42443 (local address 127.0.0.1:54890)
0812 09:47:54.542372 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0812 09:47:54.542427 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0812 09:47:54.543347 (+ 920us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:47:54.544110 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:47:54.544119 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:47:54.544681 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:47:54.545095 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:47:54.545113 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:47:54.546918 (+ 1805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:47:54.546921 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:47:54.547682 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:47:54.547691 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:47:54.548722 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:47:54.548724 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:47:54.548817 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:47:54.549471 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:47:54.549489 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:47:54.551374 (+ 1885us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:47:54.553484 (+ 2110us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:47:54.553494 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:47:54.553503 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:47:54.553744 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:47:54.554124 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:47:54.554127 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:47:54.554128 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:47:54.554248 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:47:54.554956 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:47:54.554962 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:47:54.555122 (+ 160us) client_negotiation.cc:769] Sending connection context
0812 09:47:54.555273 (+ 151us) client_negotiation.cc:240] Negotiation successful
0812 09:47:54.555544 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":151,"threads_started":1}
I20260812 09:47:54.556371 3430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:47:54.541834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54890 (local address 127.13.99.62:42443)
0812 09:47:54.542472 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0812 09:47:54.542485 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 09:47:54.542533 (+ 48us) server_negotiation.cc:407] Connection header received
0812 09:47:54.543699 (+ 1166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:47:54.543704 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:47:54.543884 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:47:54.544058 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:47:54.545281 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:47:54.546769 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:47:54.547814 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:47:54.548553 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:47:54.548766 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:47:54.551562 (+ 2796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:47:54.551568 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:47:54.551579 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:47:54.551627 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:47:54.553363 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:47:54.553883 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:47:54.553891 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:47:54.553896 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:47:54.554022 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:47:54.554337 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:47:54.554339 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:47:54.554340 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:47:54.554827 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:47:54.554968 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0812 09:47:54.555391 (+ 423us) server_negotiation.cc:299] Negotiation successful
0812 09:47:54.555682 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":67,"server-negotiator.queue_time_us":502,"thread_start_us":159,"threads_started":1}
I20260812 09:47:54.559055 3425 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.99.61" port: 43587 } has no permanent_uuid. Determining permanent_uuid...
I20260812 09:47:54.559856 3429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:47:54.559602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.61:43587 (local address 127.0.0.1:43350)
0812 09:47:54.559729 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0812 09:47:54.559798 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.61:43587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260812 09:47:54.560066 3330 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.99.61:43587: connect: Connection refused (error 111)
I20260812 09:47:54.561021 3429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:47:54.560809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.61:43587 (local address 127.0.0.1:43358)
0812 09:47:54.560919 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0812 09:47:54.560972 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.61:43587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260812 09:47:54.561285 3425 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.99.61:43587: Network error: Client connection negotiation failed: client connection to 127.13.99.61:43587: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.99.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.61.keytab.
Entry for principal kudu/127.13.99.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.61.keytab.
I20260812 09:47:54.571894 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786527905902985-13708-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786527905902985-13708-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786527905902985-13708-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786527905902985-13708-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.61:43587
--webserver_interface=127.13.99.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.61.keytab
--principal=kudu/127.13.99.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:40175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.99.62:42443,127.13.99.61:43587,127.13.99.60:38807
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 09:47:54.610522 3425 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.99.61" port: 43587 } attempt: 1
I20260812 09:47:54.611557 3429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:47:54.611232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.61:43587 (local address 127.0.0.1:43372)
0812 09:47:54.611404 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0812 09:47:54.611478 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.61:43587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260812 09:47:54.612329 3429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:47:54.612109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.61:43587 (local address 127.0.0.1:43380)
0812 09:47:54.612222 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0812 09:47:54.612274 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.61:43587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260812 09:47:54.612469 3425 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.99.61:43587: Network error: Client connection negotiation failed: client connection to 127.13.99.61:43587: connect: Connection refused (error 111)
I20260812 09:47:54.680645 3425 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.99.61" port: 43587 } attempt: 2
I20260812 09:47:54.681568 3429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:47:54.681257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.61:43587 (local address 127.0.0.1:43390)
0812 09:47:54.681414 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0812 09:47:54.681482 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.61:43587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260812 09:47:54.682268 3429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:47:54.682060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.61:43587 (local address 127.0.0.1:43400)
0812 09:47:54.682167 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0812 09:47:54.682217 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.61:43587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260812 09:47:54.682381 3425 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.99.61:43587: Network error: Client connection negotiation failed: client connection to 127.13.99.61:43587: connect: Connection refused (error 111)
W20260812 09:47:54.688114 3435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:47:54.688386 3435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:47:54.688508 3435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:47:54.692628 3435 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 09:47:54.692699 3435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:47:54.692744 3435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:47:54.692781 3435 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 09:47:54.692814 3435 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 09:47:54.698019 3435 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:40175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786527905902985-13708-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786527905902985-13708-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.13.99.62:42443,127.13.99.61:43587,127.13.99.60:38807
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.61.keytab
--principal=kudu/127.13.99.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.61:43587
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786527905902985-13708-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.13.99.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3435
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786527905902985-13708-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:47:54.699075 3435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:47:54.700093 3435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:47:54.708861 3446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:47:54.708863 3445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:47:54.709127 3435 server_base.cc:1034] running on GCE node
W20260812 09:47:54.708976 3450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:47:54.709883 3435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:47:54.711040 3435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:47:54.712234 3435 hybrid_clock.cc:648] HybridClock initialized: now 1786528074712218 us; error 32 us; skew 500 ppm
Aug 12 09:47:54 dist-test-slave-jq4t krb5kdc[3278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528074, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:47:54.715197 3435 init.cc:377] Logged in from keytab as kudu/127.13.99.61@KRBTEST.COM (short username kudu)
I20260812 09:47:54.716125 3435 webserver.cc:460] Webserver started at http://127.13.99.61:35699/ using document root <none> and password file <none>
I20260812 09:47:54.716784 3435 fs_manager.cc:359] Metadata directory not provided
I20260812 09:47:54.716879 3435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:47:54.717161 3435 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:47:54.719302 3435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786527905902985-13708-0/minicluster-data/master-1/data/instance:
uuid: "554a402446b245919090f648186bcbe2"
format_stamp: "Formatted at 2026-08-12 09:47:54 on dist-test-slave-jq4t"
server_key: "2cee098cfccc6afcf4e256b656e837dd"
server_key_iv: "2e244a6eb1a6d98677c3d55009d38cc0"
server_key_version: "encryptionkey@0"
I20260812 09:47:54.719888 3435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786527905902985-13708-0/minicluster-data/master-1/wal/instance:
uuid: "554a402446b245919090f648186bcbe2"
format_stamp: "Formatted at 2026-08-12 09:47:54 on dist-test-slave-jq4t"
server_key: "2cee098cfccc6afcf4e256b656e837dd"
server_key_iv: "2e244a6eb1a6d98677c3d55009d38cc0"
server_key_version: "encryptionkey@0"
I20260812 09:47:54.723766 3435 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 09:47:54.726323 3465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:47:54.727587 3435 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260812 09:47:54.727744 3435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786527905902985-13708-0/minicluster-data/master-1/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786527905902985-13708-0/minicluster-data/master-1/wal
uuid: "554a402446b245919090f648186bcbe2"
format_stamp: "Formatted at 2026-08-12 09:47:54 on dist-test-slave-jq4t"
server_key: "2cee098cfccc6afcf4e256b656e837dd"
server_key_iv: "2e244a6eb1a6d98677c3d55009d38cc0"
server_key_version: "encryptionkey@0"
I20260812 09:47:54.727861 3435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786527905902985-13708-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786527905902985-13708-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786527905902985-13708-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:47:54.753098 3435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:47:54.756448 3435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:47:54.756714 3435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:47:54.764008 3435 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.61:43587
I20260812 09:47:54.764094 3583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.61:43587 every 8 connection(s)
I20260812 09:47:54.765094 3435 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786527905902985-13708-0/minicluster-data/master-1/data/info.pb
I20260812 09:47:54.767998 3585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:47:54.769450 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 3435
I20260812 09:47:54.769591 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786527905902985-13708-0/minicluster-data/master-1/wal/instance
I20260812 09:47:54.770076 13708 external_mini_cluster.cc:1428] Setting key 06c423a6d6e640d6dec87c9c7cc21df7
I20260812 09:47:54.773559 3425 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.99.61" port: 43587 } attempt: 3
I20260812 09:47:54.773512 3585 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.99.62" port: 42443 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.13.99.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.99.60@KRBTEST.COM" created.
Aug 12 09:47:54 dist-test-slave-jq4t krb5kdc[3278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528074, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.61@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
Aug 12 09:47:54 dist-test-slave-jq4t krb5kdc[3278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528074, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.62@KRBTEST.COM for kudu/127.13.99.61@KRBTEST.COM
I20260812 09:47:54.789072 3429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:47:54.774184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.61:43587 (local address 127.0.0.1:43410)
0812 09:47:54.774404 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0812 09:47:54.774421 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 09:47:54.774533 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:47:54.777306 (+ 2773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:47:54.777311 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:47:54.777551 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:47:54.777683 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:47:54.777692 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:47:54.779574 (+ 1882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:47:54.779578 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:47:54.780244 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:47:54.780256 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:47:54.781034 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:47:54.781036 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:47:54.781162 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:47:54.781714 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:47:54.781726 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:47:54.784616 (+ 2890us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:47:54.786976 (+ 2360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:47:54.786979 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:47:54.786981 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:47:54.787226 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:47:54.787560 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:47:54.787563 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:47:54.787565 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:47:54.787664 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:47:54.788347 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:47:54.788353 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:47:54.788515 (+ 162us) client_negotiation.cc:769] Sending connection context
0812 09:47:54.788642 (+ 127us) client_negotiation.cc:240] Negotiation successful
0812 09:47:54.788804 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134}
I20260812 09:47:54.789940 3595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:47:54.776232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:42443 (local address 127.0.0.1:54892)
0812 09:47:54.776671 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0812 09:47:54.776721 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0812 09:47:54.777009 (+ 288us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:47:54.777551 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:47:54.777557 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:47:54.777966 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:47:54.778155 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:47:54.778165 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:47:54.778962 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:47:54.778967 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:47:54.779868 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:47:54.779879 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:47:54.780647 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:47:54.780654 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:47:54.780830 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:47:54.781570 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:47:54.781592 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:47:54.784185 (+ 2593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:47:54.786274 (+ 2089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:47:54.786286 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:47:54.786300 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:47:54.786780 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:47:54.787258 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:47:54.787264 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:47:54.787269 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:47:54.787456 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:47:54.787993 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:47:54.787999 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:47:54.788221 (+ 222us) client_negotiation.cc:769] Sending connection context
0812 09:47:54.788396 (+ 175us) client_negotiation.cc:240] Negotiation successful
0812 09:47:54.788769 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"mutex_wait_us":4,"spinlock_wait_cycles":11008,"thread_start_us":137,"threads_started":1}
I20260812 09:47:54.789955 3591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:47:54.774922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43410 (local address 127.13.99.61:43587)
0812 09:47:54.775468 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0812 09:47:54.775488 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0812 09:47:54.775541 (+ 53us) server_negotiation.cc:407] Connection header received
0812 09:47:54.776637 (+ 1096us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:47:54.776666 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:47:54.777026 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:47:54.777356 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:47:54.777873 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:47:54.779417 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:47:54.780372 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:47:54.780899 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:47:54.781027 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:47:54.784759 (+ 3732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:47:54.784767 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:47:54.784781 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:47:54.784809 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:47:54.786838 (+ 2029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:47:54.787356 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:47:54.787361 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:47:54.787362 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:47:54.787459 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:47:54.787757 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:47:54.787761 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:47:54.787763 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:47:54.788212 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:47:54.788351 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0812 09:47:54.789508 (+ 1157us) server_negotiation.cc:299] Negotiation successful
0812 09:47:54.789677 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"spinlock_wait_cycles":896,"thread_start_us":146,"threads_started":1}
I20260812 09:47:54.789709 3430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:47:54.776232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54892 (local address 127.13.99.62:42443)
0812 09:47:54.776347 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0812 09:47:54.776353 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 09:47:54.776802 (+ 449us) server_negotiation.cc:407] Connection header received
0812 09:47:54.777219 (+ 417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:47:54.777222 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:47:54.777282 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:47:54.777357 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:47:54.778301 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:47:54.778825 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:47:54.780025 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:47:54.780457 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:47:54.780619 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:47:54.784396 (+ 3777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:47:54.784403 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:47:54.784408 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:47:54.784442 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:47:54.786142 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:47:54.787007 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:47:54.787010 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:47:54.787012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:47:54.787143 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:47:54.787577 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:47:54.787583 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:47:54.787588 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:47:54.787866 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:47:54.787984 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0812 09:47:54.789443 (+ 1459us) server_negotiation.cc:299] Negotiation successful
0812 09:47:54.789578 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260812 09:47:54.791613 3585 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.99.61" port: 43587 } has no permanent_uuid. Determining permanent_uuid...
I20260812 09:47:54.792688 3425 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.99.60" port: 38807 } has no permanent_uuid. Determining permanent_uuid...
I20260812 09:47:54.793489 3429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:47:54.793179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.60:38807 (local address 127.0.0.1:39546)
0812 09:47:54.793373 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0812 09:47:54.793421 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.60:38807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260812 09:47:54.794375 3429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:47:54.794139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.60:38807 (local address 127.0.0.1:39548)
0812 09:47:54.794279 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0812 09:47:54.794325 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.60:38807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260812 09:47:54.794548 3425 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.99.60:38807: Network error: Client connection negotiation failed: client connection to 127.13.99.60:38807: connect: Connection refused (error 111)
Aug 12 09:47:54 dist-test-slave-jq4t krb5kdc[3278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528074, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.61@KRBTEST.COM for kudu/127.13.99.61@KRBTEST.COM
I20260812 09:47:54.800441 3595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:47:54.792191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.61:43587 (local address 127.0.0.1:43424)
0812 09:47:54.792281 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0812 09:47:54.792296 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 09:47:54.792374 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:47:54.792644 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:47:54.792648 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:47:54.792822 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:47:54.792936 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:47:54.792943 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:47:54.793632 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:47:54.793639 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:47:54.794533 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:47:54.794544 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:47:54.795046 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:47:54.795050 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:47:54.795175 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:47:54.795666 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:47:54.795683 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:47:54.796987 (+ 1304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:47:54.798730 (+ 1743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:47:54.798732 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:47:54.798734 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:47:54.798922 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:47:54.799297 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:47:54.799300 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:47:54.799301 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:47:54.799384 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:47:54.799918 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:47:54.799924 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:47:54.800050 (+ 126us) client_negotiation.cc:769] Sending connection context
0812 09:47:54.800155 (+ 105us) client_negotiation.cc:240] Negotiation successful
0812 09:47:54.800255 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26,"mutex_wait_us":19}
I20260812 09:47:54.800475 3591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:47:54.792277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43424 (local address 127.13.99.61:43587)
0812 09:47:54.792404 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0812 09:47:54.792410 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 09:47:54.792425 (+ 15us) server_negotiation.cc:407] Connection header received
0812 09:47:54.792460 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:47:54.792463 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:47:54.792523 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:47:54.792613 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:47:54.793038 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:47:54.793529 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:47:54.794698 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:47:54.794927 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:47:54.795049 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:47:54.797124 (+ 2075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:47:54.797127 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:47:54.797131 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:47:54.797161 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:47:54.798613 (+ 1452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:47:54.799011 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:47:54.799014 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:47:54.799015 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:47:54.799165 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:47:54.799473 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:47:54.799476 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:47:54.799477 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:47:54.799671 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:47:54.799800 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0812 09:47:54.800217 (+ 417us) server_negotiation.cc:299] Negotiation successful
0812 09:47:54.800309 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":7,"server-negotiator.queue_time_us":48}
I20260812 09:47:54.801218 3585 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.99.60" port: 38807 } has no permanent_uuid. Determining permanent_uuid...
I20260812 09:47:54.801954 3595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:47:54.801635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.60:38807 (local address 127.0.0.1:39558)
0812 09:47:54.801760 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0812 09:47:54.801870 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.60:38807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260812 09:47:54.802577 3471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.99.60:38807: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.99.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.60.keytab.
Entry for principal kudu/127.13.99.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.60.keytab.
I20260812 09:47:54.803490 3595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:47:54.803230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.60:38807 (local address 127.0.0.1:39566)
0812 09:47:54.803374 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0812 09:47:54.803430 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.60:38807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260812 09:47:54.803658 3585 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.99.60:38807: Network error: Client connection negotiation failed: client connection to 127.13.99.60:38807: connect: Connection refused (error 111)
I20260812 09:47:54.803989 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786527905902985-13708-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786527905902985-13708-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786527905902985-13708-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786527905902985-13708-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.60:38807
--webserver_interface=127.13.99.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.60.keytab
--principal=kudu/127.13.99.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:40175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.99.62:42443,127.13.99.61:43587,127.13.99.60:38807
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 09:47:54.825769 3425 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.99.60" port: 38807 } attempt: 1
I20260812 09:47:54.826632 3429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:47:54.826338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.60:38807 (local address 127.0.0.1:39574)
0812 09:47:54.826500 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0812 09:47:54.826558 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.60:38807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260812 09:47:54.827363 3429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:47:54.827172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.60:38807 (local address 127.0.0.1:39588)
0812 09:47:54.827269 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0812 09:47:54.827311 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.60:38807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260812 09:47:54.827514 3425 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.99.60:38807: Network error: Client connection negotiation failed: client connection to 127.13.99.60:38807: connect: Connection refused (error 111)
I20260812 09:47:54.852924 3585 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.99.60" port: 38807 } attempt: 1
I20260812 09:47:54.853848 3595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:47:54.853511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.60:38807 (local address 127.0.0.1:39598)
0812 09:47:54.853684 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0812 09:47:54.853755 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.60:38807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260812 09:47:54.854664 3595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:47:54.854424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.60:38807 (local address 127.0.0.1:39602)
0812 09:47:54.854545 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0812 09:47:54.854598 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.60:38807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260812 09:47:54.854825 3585 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.99.60:38807: Network error: Client connection negotiation failed: client connection to 127.13.99.60:38807: connect: Connection refused (error 111)
I20260812 09:47:54.902689 3425 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.99.60" port: 38807 } attempt: 2
I20260812 09:47:54.903640 3429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:47:54.903330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.60:38807 (local address 127.0.0.1:39618)
0812 09:47:54.903502 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0812 09:47:54.903561 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.60:38807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260812 09:47:54.904379 3429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:47:54.904173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.60:38807 (local address 127.0.0.1:39632)
0812 09:47:54.904282 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0812 09:47:54.904326 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.60:38807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260812 09:47:54.904567 3425 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.99.60:38807: Network error: Client connection negotiation failed: client connection to 127.13.99.60:38807: connect: Connection refused (error 111)
W20260812 09:47:54.919029 3598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:47:54.919275 3598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:47:54.919315 3598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260812 09:47:54.922997 3585 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.99.60" port: 38807 } attempt: 2
W20260812 09:47:54.923385 3598 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 09:47:54.923442 3598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:47:54.923471 3598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:47:54.923493 3598 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 09:47:54.923527 3598 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 09:47:54.923808 3595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:47:54.923484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.60:38807 (local address 127.0.0.1:39634)
0812 09:47:54.923668 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0812 09:47:54.923730 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.60:38807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260812 09:47:54.924578 3595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:47:54.924335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.60:38807 (local address 127.0.0.1:39644)
0812 09:47:54.924431 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0812 09:47:54.924489 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.60:38807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260812 09:47:54.924722 3585 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.99.60:38807: Network error: Client connection negotiation failed: client connection to 127.13.99.60:38807: connect: Connection refused (error 111)
I20260812 09:47:54.929019 3598 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:40175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786527905902985-13708-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786527905902985-13708-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.13.99.62:42443,127.13.99.61:43587,127.13.99.60:38807
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.60.keytab
--principal=kudu/127.13.99.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.60:38807
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786527905902985-13708-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.13.99.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3598
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786527905902985-13708-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:47:54.930044 3598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:47:54.931126 3598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:47:54.939653 3607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:47:54.939644 3605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:47:54.939785 3598 server_base.cc:1034] running on GCE node
W20260812 09:47:54.939639 3609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:47:54.940591 3598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:47:54.941601 3598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:47:54.942762 3598 hybrid_clock.cc:648] HybridClock initialized: now 1786528074942751 us; error 24 us; skew 500 ppm
Aug 12 09:47:54 dist-test-slave-jq4t krb5kdc[3278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528074, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:47:54.945711 3598 init.cc:377] Logged in from keytab as kudu/127.13.99.60@KRBTEST.COM (short username kudu)
I20260812 09:47:54.946619 3598 webserver.cc:460] Webserver started at http://127.13.99.60:34879/ using document root <none> and password file <none>
I20260812 09:47:54.947176 3598 fs_manager.cc:359] Metadata directory not provided
I20260812 09:47:54.947237 3598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:47:54.947487 3598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:47:54.949575 3598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786527905902985-13708-0/minicluster-data/master-2/data/instance:
uuid: "bbcd4ccf923040a09d2dc68313713137"
format_stamp: "Formatted at 2026-08-12 09:47:54 on dist-test-slave-jq4t"
server_key: "9da764e03f97b9ec1700f2cd0a127885"
server_key_iv: "fe71b34034fc4bad60e2e7b8e02e5a61"
server_key_version: "encryptionkey@0"
I20260812 09:47:54.950186 3598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786527905902985-13708-0/minicluster-data/master-2/wal/instance:
uuid: "bbcd4ccf923040a09d2dc68313713137"
format_stamp: "Formatted at 2026-08-12 09:47:54 on dist-test-slave-jq4t"
server_key: "9da764e03f97b9ec1700f2cd0a127885"
server_key_iv: "fe71b34034fc4bad60e2e7b8e02e5a61"
server_key_version: "encryptionkey@0"
I20260812 09:47:54.953980 3598 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 09:47:54.956406 3621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:47:54.957688 3598 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 09:47:54.957831 3598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786527905902985-13708-0/minicluster-data/master-2/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786527905902985-13708-0/minicluster-data/master-2/wal
uuid: "bbcd4ccf923040a09d2dc68313713137"
format_stamp: "Formatted at 2026-08-12 09:47:54 on dist-test-slave-jq4t"
server_key: "9da764e03f97b9ec1700f2cd0a127885"
server_key_iv: "fe71b34034fc4bad60e2e7b8e02e5a61"
server_key_version: "encryptionkey@0"
I20260812 09:47:54.957942 3598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786527905902985-13708-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786527905902985-13708-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786527905902985-13708-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:47:54.971144 3598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:47:54.974471 3598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:47:54.974676 3598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:47:54.982090 3598 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.60:38807
I20260812 09:47:54.982102 3724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.60:38807 every 8 connection(s)
I20260812 09:47:54.983064 3598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786527905902985-13708-0/minicluster-data/master-2/data/info.pb
I20260812 09:47:54.985993 3725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:47:54.990175 3725 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.99.62" port: 42443 } has no permanent_uuid. Determining permanent_uuid...
I20260812 09:47:54.990993 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 3598
I20260812 09:47:54.991155 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786527905902985-13708-0/minicluster-data/master-2/wal/instance
I20260812 09:47:54.991452 13708 external_mini_cluster.cc:1428] Setting key b78d4eca15bd93c63d2ad8e7203852af
Aug 12 09:47:54 dist-test-slave-jq4t krb5kdc[3278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528074, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.60@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
Aug 12 09:47:54 dist-test-slave-jq4t krb5kdc[3278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528074, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.99.60@KRBTEST.COM
I20260812 09:47:55.003711 3425 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.99.60" port: 38807 } attempt: 3
I20260812 09:47:55.004789 3430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:47:54.992213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54904 (local address 127.13.99.62:42443)
0812 09:47:54.992353 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0812 09:47:54.992358 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:47:54.993372 (+ 1014us) server_negotiation.cc:407] Connection header received
0812 09:47:54.994171 (+ 799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:47:54.994174 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:47:54.994223 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:47:54.994338 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:47:54.995427 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:47:54.995914 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:47:54.997043 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:47:54.997286 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:47:54.997369 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:47:55.000254 (+ 2885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:47:55.000257 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:47:55.000259 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:47:55.000285 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:47:55.002116 (+ 1831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:47:55.002762 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:47:55.002766 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:47:55.002768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:47:55.002868 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:47:55.003299 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:47:55.003303 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:47:55.003304 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:47:55.003595 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:47:55.003675 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0812 09:47:55.004457 (+ 782us) server_negotiation.cc:299] Negotiation successful
0812 09:47:55.004623 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260812 09:47:55.006523 3733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:47:54.992693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39656 (local address 127.13.99.60:38807)
0812 09:47:54.993125 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0812 09:47:54.993139 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 09:47:54.993213 (+ 74us) server_negotiation.cc:407] Connection header received
0812 09:47:54.993921 (+ 708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:47:54.993939 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:47:54.994184 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:47:54.994371 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:47:54.995128 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:47:54.996145 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:47:54.997028 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:47:54.997493 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:47:54.997697 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:47:55.000731 (+ 3034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:47:55.000739 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:47:55.000773 (+ 34us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:47:55.000803 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:47:55.002770 (+ 1967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:47:55.003324 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:47:55.003330 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:47:55.003331 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:47:55.003413 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:47:55.003788 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:47:55.003791 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:47:55.003793 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:47:55.004164 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:47:55.005082 (+ 918us) server_negotiation.cc:1035] Waiting for connection context
0812 09:47:55.005211 (+ 129us) server_negotiation.cc:299] Negotiation successful
0812 09:47:55.005579 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":134,"server-negotiator.queue_time_us":288,"thread_start_us":134,"threads_started":1}
I20260812 09:47:55.006531 3736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:47:54.992693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:42443 (local address 127.0.0.1:54904)
0812 09:47:54.993212 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0812 09:47:54.993261 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0812 09:47:54.993940 (+ 679us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:47:54.994521 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:47:54.994526 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:47:54.994927 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:47:54.995272 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:47:54.995285 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:47:54.996023 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:47:54.996027 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:47:54.996913 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:47:54.996923 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:47:54.997428 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:47:54.997433 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:47:54.997699 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:47:54.998479 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:47:54.998495 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:47:55.000085 (+ 1590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:47:55.002255 (+ 2170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:47:55.002263 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:47:55.002272 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:47:55.002646 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:47:55.002983 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:47:55.002986 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:47:55.002988 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:47:55.003177 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:47:55.003723 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:47:55.003732 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:47:55.004144 (+ 412us) client_negotiation.cc:769] Sending connection context
0812 09:47:55.005215 (+ 1071us) client_negotiation.cc:240] Negotiation successful
0812 09:47:55.005663 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"mutex_wait_us":204,"spinlock_wait_cycles":17536,"thread_start_us":187,"threads_started":1}
I20260812 09:47:55.008425 3725 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.99.61" port: 43587 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 09:47:55 dist-test-slave-jq4t krb5kdc[3278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528074, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.62@KRBTEST.COM for kudu/127.13.99.60@KRBTEST.COM
Aug 12 09:47:55 dist-test-slave-jq4t krb5kdc[3278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528074, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
I20260812 09:47:55.015390 3429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:47:55.004286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.60:38807 (local address 127.0.0.1:39666)
0812 09:47:55.004661 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0812 09:47:55.004680 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 09:47:55.004761 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:47:55.005050 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:47:55.005053 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:47:55.005238 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:47:55.005356 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:47:55.005364 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:47:55.006470 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:47:55.006474 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:47:55.007125 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:47:55.007136 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:47:55.007695 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:47:55.007699 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:47:55.007811 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:47:55.008320 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:47:55.008334 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:47:55.009928 (+ 1594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:47:55.013684 (+ 3756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:47:55.013689 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:47:55.013695 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:47:55.013904 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:47:55.014384 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:47:55.014387 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:47:55.014388 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:47:55.014479 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:47:55.014871 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:47:55.014874 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:47:55.014972 (+ 98us) client_negotiation.cc:769] Sending connection context
0812 09:47:55.015083 (+ 111us) client_negotiation.cc:240] Negotiation successful
0812 09:47:55.015222 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284}
I20260812 09:47:55.015421 3738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:47:55.004324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39666 (local address 127.13.99.60:38807)
0812 09:47:55.004701 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0812 09:47:55.004710 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 09:47:55.004724 (+ 14us) server_negotiation.cc:407] Connection header received
0812 09:47:55.004882 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:47:55.004885 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:47:55.004937 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:47:55.005025 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:47:55.005487 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:47:55.006264 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:47:55.007249 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:47:55.007569 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:47:55.007670 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:47:55.010078 (+ 2408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:47:55.010082 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:47:55.010085 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:47:55.010113 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:47:55.013553 (+ 3440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:47:55.014015 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:47:55.014017 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:47:55.014018 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:47:55.014098 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:47:55.014582 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:47:55.014586 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:47:55.014588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:47:55.014769 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:47:55.014877 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0812 09:47:55.015142 (+ 265us) server_negotiation.cc:299] Negotiation successful
0812 09:47:55.015253 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":158,"threads_started":1}
Aug 12 09:47:55 dist-test-slave-jq4t krb5kdc[3278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528074, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.60@KRBTEST.COM for kudu/127.13.99.61@KRBTEST.COM
I20260812 09:47:55.016260 3585 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.99.60" port: 38807 } attempt: 3
I20260812 09:47:55.020224 3430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:47:55.009627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54920 (local address 127.13.99.62:42443)
0812 09:47:55.009730 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0812 09:47:55.009734 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:47:55.009747 (+ 13us) server_negotiation.cc:407] Connection header received
0812 09:47:55.009782 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:47:55.009784 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:47:55.009826 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:47:55.009919 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:47:55.010875 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:47:55.011554 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:47:55.012649 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:47:55.013019 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:47:55.013136 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:47:55.016593 (+ 3457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:47:55.016601 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:47:55.016606 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:47:55.016633 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:47:55.018522 (+ 1889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:47:55.018974 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:47:55.018978 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:47:55.018980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:47:55.019071 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:47:55.019386 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:47:55.019390 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:47:55.019391 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:47:55.019588 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:47:55.019776 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0812 09:47:55.019975 (+ 199us) server_negotiation.cc:299] Negotiation successful
0812 09:47:55.020084 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260812 09:47:55.020401 3425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f61b4a745f74e73ae77dfa285a9d849: Bootstrap starting.
I20260812 09:47:55.022861 3591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:47:55.009416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43438 (local address 127.13.99.61:43587)
0812 09:47:55.009558 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0812 09:47:55.009564 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 09:47:55.010182 (+ 618us) server_negotiation.cc:407] Connection header received
0812 09:47:55.010291 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:47:55.010293 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:47:55.010354 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:47:55.010441 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:47:55.011373 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:47:55.012076 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:47:55.013216 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:47:55.013573 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:47:55.013702 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:47:55.016498 (+ 2796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:47:55.016504 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:47:55.016507 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:47:55.016538 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:47:55.018522 (+ 1984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:47:55.020059 (+ 1537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:47:55.020064 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:47:55.020067 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:47:55.020166 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:47:55.020623 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:47:55.020626 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:47:55.020628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:47:55.020831 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:47:55.020891 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0812 09:47:55.022540 (+ 1649us) server_negotiation.cc:299] Negotiation successful
0812 09:47:55.022690 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260812 09:47:55.023355 3736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:47:55.009138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.61:43587 (local address 127.0.0.1:43438)
0812 09:47:55.010098 (+ 960us) negotiation.cc:107] Waiting for socket to connect
0812 09:47:55.010109 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 09:47:55.010205 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:47:55.010456 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:47:55.010458 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:47:55.011131 (+ 673us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:47:55.011248 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:47:55.011255 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:47:55.012216 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:47:55.012219 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:47:55.013037 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:47:55.013048 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:47:55.013880 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:47:55.013884 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:47:55.014007 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:47:55.014396 (+ 389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:47:55.014417 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:47:55.016273 (+ 1856us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:47:55.018930 (+ 2657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:47:55.018934 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:47:55.018935 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:47:55.019176 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:47:55.020313 (+ 1137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:47:55.020316 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:47:55.020317 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:47:55.020440 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:47:55.020992 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:47:55.020998 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:47:55.021097 (+ 99us) client_negotiation.cc:769] Sending connection context
0812 09:47:55.023053 (+ 1956us) client_negotiation.cc:240] Negotiation successful
0812 09:47:55.023201 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":885}
I20260812 09:47:55.023799 3425 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4f61b4a745f74e73ae77dfa285a9d849: Neither blocks nor log segments found. Creating new log.
I20260812 09:47:55.024339 3725 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.99.60" port: 38807 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 09:47:55 dist-test-slave-jq4t krb5kdc[3278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528074, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.61@KRBTEST.COM for kudu/127.13.99.60@KRBTEST.COM
I20260812 09:47:55.025458 3425 log.cc:824] T 00000000000000000000000000000000 P 4f61b4a745f74e73ae77dfa285a9d849: Log is configured to *not* fsync() on all Append() calls
Aug 12 09:47:55 dist-test-slave-jq4t krb5kdc[3278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528074, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.99.61@KRBTEST.COM
I20260812 09:47:55.028286 3425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f61b4a745f74e73ae77dfa285a9d849: No bootstrap required, opened a new log
I20260812 09:47:55.030815 3595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:47:55.016687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.60:38807 (local address 127.0.0.1:39670)
0812 09:47:55.017230 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0812 09:47:55.017243 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 09:47:55.017304 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:47:55.018238 (+ 934us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:47:55.018241 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:47:55.018675 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:47:55.018780 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:47:55.018790 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:47:55.021248 (+ 2458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:47:55.021252 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:47:55.021938 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:47:55.021947 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:47:55.023818 (+ 1871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:47:55.023824 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:47:55.023912 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:47:55.024336 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:47:55.024350 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:47:55.025850 (+ 1500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:47:55.028755 (+ 2905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:47:55.028759 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:47:55.028761 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:47:55.029264 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:47:55.029591 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:47:55.029594 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:47:55.029596 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:47:55.029692 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:47:55.030164 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:47:55.030170 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:47:55.030291 (+ 121us) client_negotiation.cc:769] Sending connection context
0812 09:47:55.030426 (+ 135us) client_negotiation.cc:240] Negotiation successful
0812 09:47:55.030567 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474}
I20260812 09:47:55.030822 3738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:47:55.017414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39670 (local address 127.13.99.60:38807)
0812 09:47:55.017972 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0812 09:47:55.017978 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 09:47:55.017995 (+ 17us) server_negotiation.cc:407] Connection header received
0812 09:47:55.018031 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:47:55.018034 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:47:55.018088 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:47:55.018458 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:47:55.019282 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:47:55.019929 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:47:55.022056 (+ 2127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:47:55.022340 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:47:55.024127 (+ 1787us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:47:55.026031 (+ 1904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:47:55.026035 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:47:55.026038 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:47:55.026061 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:47:55.028097 (+ 2036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:47:55.029388 (+ 1291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:47:55.029391 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:47:55.029393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:47:55.029479 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:47:55.029821 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:47:55.029824 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:47:55.029826 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:47:55.030030 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:47:55.030155 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0812 09:47:55.030488 (+ 333us) server_negotiation.cc:299] Negotiation successful
0812 09:47:55.030610 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473}
Aug 12 09:47:55 dist-test-slave-jq4t krb5kdc[3278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528074, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.60@KRBTEST.COM for kudu/127.13.99.60@KRBTEST.COM
I20260812 09:47:55.033013 3741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:47:55.021722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43440 (local address 127.13.99.61:43587)
0812 09:47:55.022069 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0812 09:47:55.022080 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 09:47:55.022097 (+ 17us) server_negotiation.cc:407] Connection header received
0812 09:47:55.022136 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:47:55.022140 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:47:55.022183 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:47:55.022274 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:47:55.023037 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:47:55.023775 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:47:55.024848 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:47:55.025265 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:47:55.025376 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:47:55.028778 (+ 3402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:47:55.028782 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:47:55.028785 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:47:55.028812 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:47:55.030863 (+ 2051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:47:55.031404 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:47:55.031407 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:47:55.031409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:47:55.031504 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:47:55.031848 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:47:55.031854 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:47:55.031857 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:47:55.032113 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:47:55.032303 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0812 09:47:55.032711 (+ 408us) server_negotiation.cc:299] Negotiation successful
0812 09:47:55.032828 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":124,"threads_started":1}
I20260812 09:47:55.034541 3585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 554a402446b245919090f648186bcbe2: Bootstrap starting.
I20260812 09:47:55.036262 3736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:47:55.024849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.60:38807 (local address 127.0.0.1:39680)
0812 09:47:55.026371 (+ 1522us) negotiation.cc:107] Waiting for socket to connect
0812 09:47:55.026386 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 09:47:55.026499 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:47:55.026841 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:47:55.026845 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:47:55.027009 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:47:55.027125 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:47:55.027133 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:47:55.028097 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:47:55.028101 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:47:55.028944 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:47:55.028952 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:47:55.029657 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:47:55.029662 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:47:55.029759 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:47:55.031018 (+ 1259us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:47:55.031035 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:47:55.032574 (+ 1539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:47:55.034651 (+ 2077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:47:55.034655 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:47:55.034656 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:47:55.034889 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:47:55.035196 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:47:55.035199 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:47:55.035200 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:47:55.035278 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:47:55.035714 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:47:55.035717 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:47:55.035804 (+ 87us) client_negotiation.cc:769] Sending connection context
0812 09:47:55.035910 (+ 106us) client_negotiation.cc:240] Negotiation successful
0812 09:47:55.036067 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1442}
I20260812 09:47:55.036295 3733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:47:55.025068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39680 (local address 127.13.99.60:38807)
0812 09:47:55.025256 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0812 09:47:55.025260 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:47:55.026461 (+ 1201us) server_negotiation.cc:407] Connection header received
0812 09:47:55.026614 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:47:55.026619 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:47:55.026662 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:47:55.026736 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:47:55.027259 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:47:55.027954 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:47:55.029090 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:47:55.029413 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:47:55.029525 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:47:55.032699 (+ 3174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:47:55.032702 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:47:55.032705 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:47:55.032727 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:47:55.034538 (+ 1811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:47:55.035011 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:47:55.035014 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:47:55.035016 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:47:55.035092 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:47:55.035385 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:47:55.035390 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:47:55.035391 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:47:55.035610 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:47:55.035724 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0812 09:47:55.035980 (+ 256us) server_negotiation.cc:299] Negotiation successful
0812 09:47:55.036137 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":119}
I20260812 09:47:55.036865 13708 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 09:47:55.037460 3585 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 554a402446b245919090f648186bcbe2: Neither blocks nor log segments found. Creating new log.
I20260812 09:47:55.038326 3585 log.cc:824] T 00000000000000000000000000000000 P 554a402446b245919090f648186bcbe2: Log is configured to *not* fsync() on all Append() calls
I20260812 09:47:55.039211 3725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbcd4ccf923040a09d2dc68313713137: Bootstrap starting.
I20260812 09:47:55.040307 3425 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4f61b4a745f74e73ae77dfa285a9d849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f61b4a745f74e73ae77dfa285a9d849" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 42443 } } peers { permanent_uuid: "554a402446b245919090f648186bcbe2" member_type: VOTER last_known_addr { host: "127.13.99.61" port: 43587 } } peers { permanent_uuid: "bbcd4ccf923040a09d2dc68313713137" member_type: VOTER last_known_addr { host: "127.13.99.60" port: 38807 } }
I20260812 09:47:55.040794 3425 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4f61b4a745f74e73ae77dfa285a9d849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:47:55.040912 3425 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4f61b4a745f74e73ae77dfa285a9d849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f61b4a745f74e73ae77dfa285a9d849, State: Initialized, Role: FOLLOWER
I20260812 09:47:55.041491 3425 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f61b4a745f74e73ae77dfa285a9d849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f61b4a745f74e73ae77dfa285a9d849" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 42443 } } peers { permanent_uuid: "554a402446b245919090f648186bcbe2" member_type: VOTER last_known_addr { host: "127.13.99.61" port: 43587 } } peers { permanent_uuid: "bbcd4ccf923040a09d2dc68313713137" member_type: VOTER last_known_addr { host: "127.13.99.60" port: 38807 } }
I20260812 09:47:55.042116 3725 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bbcd4ccf923040a09d2dc68313713137: Neither blocks nor log segments found. Creating new log.
I20260812 09:47:55.042992 3425 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4f61b4a745f74e73ae77dfa285a9d849 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 09:47:55.043008 3725 log.cc:824] T 00000000000000000000000000000000 P bbcd4ccf923040a09d2dc68313713137: Log is configured to *not* fsync() on all Append() calls
I20260812 09:47:55.045217 3725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbcd4ccf923040a09d2dc68313713137: No bootstrap required, opened a new log
I20260812 09:47:55.048838 3585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 554a402446b245919090f648186bcbe2: No bootstrap required, opened a new log
I20260812 09:47:55.050575 3430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:47:55.039668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54934 (local address 127.13.99.62:42443)
0812 09:47:55.039798 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0812 09:47:55.039805 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 09:47:55.039821 (+ 16us) server_negotiation.cc:407] Connection header received
0812 09:47:55.039861 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:47:55.039864 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:47:55.039913 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:47:55.039992 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:47:55.041012 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:47:55.041814 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:47:55.042905 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:47:55.043238 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:47:55.043314 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:47:55.045268 (+ 1954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:47:55.045275 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:47:55.045279 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:47:55.045314 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:47:55.047339 (+ 2025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:47:55.049186 (+ 1847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:47:55.049190 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:47:55.049192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:47:55.049284 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:47:55.049610 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:47:55.049613 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:47:55.049615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:47:55.049829 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:47:55.049943 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0812 09:47:55.050288 (+ 345us) server_negotiation.cc:299] Negotiation successful
0812 09:47:55.050407 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260812 09:47:55.050487 3725 raft_consensus.cc:348] T 00000000000000000000000000000000 P bbcd4ccf923040a09d2dc68313713137 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f61b4a745f74e73ae77dfa285a9d849" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 42443 } } peers { permanent_uuid: "554a402446b245919090f648186bcbe2" member_type: VOTER last_known_addr { host: "127.13.99.61" port: 43587 } } peers { permanent_uuid: "bbcd4ccf923040a09d2dc68313713137" member_type: VOTER last_known_addr { host: "127.13.99.60" port: 38807 } }
I20260812 09:47:55.050729 3725 raft_consensus.cc:374] T 00000000000000000000000000000000 P bbcd4ccf923040a09d2dc68313713137 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:47:55.050822 3725 raft_consensus.cc:729] T 00000000000000000000000000000000 P bbcd4ccf923040a09d2dc68313713137 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbcd4ccf923040a09d2dc68313713137, State: Initialized, Role: FOLLOWER
I20260812 09:47:55.051378 3725 consensus_queue.cc:260] T 00000000000000000000000000000000 P bbcd4ccf923040a09d2dc68313713137 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f61b4a745f74e73ae77dfa285a9d849" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 42443 } } peers { permanent_uuid: "554a402446b245919090f648186bcbe2" member_type: VOTER last_known_addr { host: "127.13.99.61" port: 43587 } } peers { permanent_uuid: "bbcd4ccf923040a09d2dc68313713137" member_type: VOTER last_known_addr { host: "127.13.99.60" port: 38807 } }
I20260812 09:47:55.052842 3725 sys_catalog.cc:564] T 00000000000000000000000000000000 P bbcd4ccf923040a09d2dc68313713137 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 09:47:55.048441 3298 master_runner.cc:186] Error getting master registration for 127.13.99.61:43587: OK, instance_id { permanent_uuid: "554a402446b245919090f648186bcbe2" instance_seqno: 1786528074761964 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 09:47:55.052974 3298 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 09:47:55.053992 3585 raft_consensus.cc:348] T 00000000000000000000000000000000 P 554a402446b245919090f648186bcbe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f61b4a745f74e73ae77dfa285a9d849" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 42443 } } peers { permanent_uuid: "554a402446b245919090f648186bcbe2" member_type: VOTER last_known_addr { host: "127.13.99.61" port: 43587 } } peers { permanent_uuid: "bbcd4ccf923040a09d2dc68313713137" member_type: VOTER last_known_addr { host: "127.13.99.60" port: 38807 } }
I20260812 09:47:55.054303 3585 raft_consensus.cc:374] T 00000000000000000000000000000000 P 554a402446b245919090f648186bcbe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:47:55.054420 3585 raft_consensus.cc:729] T 00000000000000000000000000000000 P 554a402446b245919090f648186bcbe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 554a402446b245919090f648186bcbe2, State: Initialized, Role: FOLLOWER
I20260812 09:47:55.054859 3733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:47:55.039858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39686 (local address 127.13.99.60:38807)
0812 09:47:55.039960 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0812 09:47:55.039966 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 09:47:55.039979 (+ 13us) server_negotiation.cc:407] Connection header received
0812 09:47:55.040010 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:47:55.040013 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:47:55.040046 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:47:55.040115 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:47:55.041786 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:47:55.042552 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:47:55.043621 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:47:55.043997 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:47:55.044096 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:47:55.046235 (+ 2139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:47:55.046241 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:47:55.046245 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:47:55.046281 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:47:55.048704 (+ 2423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:47:55.053498 (+ 4794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:47:55.053502 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:47:55.053504 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:47:55.053602 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:47:55.053899 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:47:55.053903 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:47:55.053905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:47:55.054129 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:47:55.054252 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0812 09:47:55.054595 (+ 343us) server_negotiation.cc:299] Negotiation successful
0812 09:47:55.054712 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260812 09:47:55.056306 3782 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbcd4ccf923040a09d2dc68313713137 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f61b4a745f74e73ae77dfa285a9d849" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 42443 } } peers { permanent_uuid: "554a402446b245919090f648186bcbe2" member_type: VOTER last_known_addr { host: "127.13.99.61" port: 43587 } } peers { permanent_uuid: "bbcd4ccf923040a09d2dc68313713137" member_type: VOTER last_known_addr { host: "127.13.99.60" port: 38807 } } }
I20260812 09:47:55.056526 3782 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbcd4ccf923040a09d2dc68313713137 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 09:47:55.055156 3585 consensus_queue.cc:260] T 00000000000000000000000000000000 P 554a402446b245919090f648186bcbe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f61b4a745f74e73ae77dfa285a9d849" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 42443 } } peers { permanent_uuid: "554a402446b245919090f648186bcbe2" member_type: VOTER last_known_addr { host: "127.13.99.61" port: 43587 } } peers { permanent_uuid: "bbcd4ccf923040a09d2dc68313713137" member_type: VOTER last_known_addr { host: "127.13.99.60" port: 38807 } }
I20260812 09:47:55.059334 3585 sys_catalog.cc:564] T 00000000000000000000000000000000 P 554a402446b245919090f648186bcbe2 [sys.catalog]: configured and running, proceeding with master startup.
W20260812 09:47:55.059855 3802 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bbcd4ccf923040a09d2dc68313713137: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 09:47:55.059981 3802 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 09:47:55.060767 3741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:47:55.040410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43448 (local address 127.13.99.61:43587)
0812 09:47:55.040617 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0812 09:47:55.040622 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:47:55.051594 (+ 10972us) server_negotiation.cc:407] Connection header received
0812 09:47:55.051812 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:47:55.051815 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:47:55.051863 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:47:55.051916 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:47:55.052761 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:47:55.053497 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:47:55.054586 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:47:55.054932 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:47:55.055015 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:47:55.056744 (+ 1729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:47:55.056747 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:47:55.056749 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:47:55.056775 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:47:55.058559 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:47:55.059025 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:47:55.059030 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:47:55.059032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:47:55.059133 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:47:55.059440 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:47:55.059443 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:47:55.059445 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:47:55.059669 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:47:55.059803 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0812 09:47:55.060452 (+ 649us) server_negotiation.cc:299] Negotiation successful
0812 09:47:55.060633 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
I20260812 09:47:55.060113 3598 master_runner.cc:186] Error getting master registration for 127.13.99.61:43587: OK, instance_id { permanent_uuid: "554a402446b245919090f648186bcbe2" instance_seqno: 1786528074761964 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 09:47:55.063691 3598 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 09:47:55.064702 3805 sys_catalog.cc:455] T 00000000000000000000000000000000 P 554a402446b245919090f648186bcbe2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f61b4a745f74e73ae77dfa285a9d849" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 42443 } } peers { permanent_uuid: "554a402446b245919090f648186bcbe2" member_type: VOTER last_known_addr { host: "127.13.99.61" port: 43587 } } peers { permanent_uuid: "bbcd4ccf923040a09d2dc68313713137" member_type: VOTER last_known_addr { host: "127.13.99.60" port: 38807 } } }
I20260812 09:47:55.064714 3761 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f61b4a745f74e73ae77dfa285a9d849 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f61b4a745f74e73ae77dfa285a9d849" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 42443 } } peers { permanent_uuid: "554a402446b245919090f648186bcbe2" member_type: VOTER last_known_addr { host: "127.13.99.61" port: 43587 } } peers { permanent_uuid: "bbcd4ccf923040a09d2dc68313713137" member_type: VOTER last_known_addr { host: "127.13.99.60" port: 38807 } } }
I20260812 09:47:55.064858 3761 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f61b4a745f74e73ae77dfa285a9d849 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 09:47:55.064889 3805 sys_catalog.cc:458] T 00000000000000000000000000000000 P 554a402446b245919090f648186bcbe2 [sys.catalog]: This master's current role is: FOLLOWER
W20260812 09:47:55.066807 3780 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4f61b4a745f74e73ae77dfa285a9d849: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260812 09:47:55.067515 3435 master_runner.cc:263] No leader master found from master 554a402446b245919090f648186bcbe2
I20260812 09:47:55.067574 3435 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260812 09:47:55.066920 3780 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260812 09:47:55.069321 3826 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 554a402446b245919090f648186bcbe2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 09:47:55.069396 3826 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260812 09:47:55.226096 3631 tablet.cc:2369] T 00000000000000000000000000000000 P bbcd4ccf923040a09d2dc68313713137: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 09:47:55.258409 3477 tablet.cc:2369] T 00000000000000000000000000000000 P 554a402446b245919090f648186bcbe2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 09:47:55.275429 3337 tablet.cc:2369] T 00000000000000000000000000000000 P 4f61b4a745f74e73ae77dfa285a9d849: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 09:47:55.374593 3805 raft_consensus.cc:482] T 00000000000000000000000000000000 P 554a402446b245919090f648186bcbe2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 09:47:55.374869 3805 raft_consensus.cc:504] T 00000000000000000000000000000000 P 554a402446b245919090f648186bcbe2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f61b4a745f74e73ae77dfa285a9d849" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 42443 } } peers { permanent_uuid: "554a402446b245919090f648186bcbe2" member_type: VOTER last_known_addr { host: "127.13.99.61" port: 43587 } } peers { permanent_uuid: "bbcd4ccf923040a09d2dc68313713137" member_type: VOTER last_known_addr { host: "127.13.99.60" port: 38807 } }
I20260812 09:47:55.375909 3805 leader_election.cc:288] T 00000000000000000000000000000000 P 554a402446b245919090f648186bcbe2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4f61b4a745f74e73ae77dfa285a9d849 (127.13.99.62:42443), bbcd4ccf923040a09d2dc68313713137 (127.13.99.60:38807)
I20260812 09:47:55.376566 3371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "554a402446b245919090f648186bcbe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f61b4a745f74e73ae77dfa285a9d849" is_pre_election: true
I20260812 09:47:55.376679 3666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "554a402446b245919090f648186bcbe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbcd4ccf923040a09d2dc68313713137" is_pre_election: true
I20260812 09:47:55.376922 3371 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4f61b4a745f74e73ae77dfa285a9d849 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 554a402446b245919090f648186bcbe2 in term 0.
I20260812 09:47:55.377048 3666 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bbcd4ccf923040a09d2dc68313713137 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 554a402446b245919090f648186bcbe2 in term 0.
I20260812 09:47:55.377463 3471 leader_election.cc:302] T 00000000000000000000000000000000 P 554a402446b245919090f648186bcbe2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f61b4a745f74e73ae77dfa285a9d849, 554a402446b245919090f648186bcbe2; no voters:
I20260812 09:47:55.377764 3805 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 554a402446b245919090f648186bcbe2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 09:47:55.377874 3805 raft_consensus.cc:482] T 00000000000000000000000000000000 P 554a402446b245919090f648186bcbe2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 09:47:55.377933 3805 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 554a402446b245919090f648186bcbe2 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:47:55.379112 3805 raft_consensus.cc:504] T 00000000000000000000000000000000 P 554a402446b245919090f648186bcbe2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f61b4a745f74e73ae77dfa285a9d849" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 42443 } } peers { permanent_uuid: "554a402446b245919090f648186bcbe2" member_type: VOTER last_known_addr { host: "127.13.99.61" port: 43587 } } peers { permanent_uuid: "bbcd4ccf923040a09d2dc68313713137" member_type: VOTER last_known_addr { host: "127.13.99.60" port: 38807 } }
I20260812 09:47:55.379545 3805 leader_election.cc:288] T 00000000000000000000000000000000 P 554a402446b245919090f648186bcbe2 [CANDIDATE]: Term 1 election: Requested vote from peers 4f61b4a745f74e73ae77dfa285a9d849 (127.13.99.62:42443), bbcd4ccf923040a09d2dc68313713137 (127.13.99.60:38807)
I20260812 09:47:55.379995 3371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "554a402446b245919090f648186bcbe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f61b4a745f74e73ae77dfa285a9d849"
I20260812 09:47:55.380148 3371 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4f61b4a745f74e73ae77dfa285a9d849 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:47:55.380158 3666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "554a402446b245919090f648186bcbe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbcd4ccf923040a09d2dc68313713137"
I20260812 09:47:55.380314 3666 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bbcd4ccf923040a09d2dc68313713137 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:47:55.381913 3666 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bbcd4ccf923040a09d2dc68313713137 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 554a402446b245919090f648186bcbe2 in term 1.
I20260812 09:47:55.381917 3371 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4f61b4a745f74e73ae77dfa285a9d849 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 554a402446b245919090f648186bcbe2 in term 1.
I20260812 09:47:55.387580 3471 leader_election.cc:302] T 00000000000000000000000000000000 P 554a402446b245919090f648186bcbe2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f61b4a745f74e73ae77dfa285a9d849, 554a402446b245919090f648186bcbe2; no voters:
I20260812 09:47:55.387998 3805 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 554a402446b245919090f648186bcbe2 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 09:47:55.388456 3805 raft_consensus.cc:686] T 00000000000000000000000000000000 P 554a402446b245919090f648186bcbe2 [term 1 LEADER]: Becoming Leader. State: Replica: 554a402446b245919090f648186bcbe2, State: Running, Role: LEADER
I20260812 09:47:55.389125 3805 consensus_queue.cc:237] T 00000000000000000000000000000000 P 554a402446b245919090f648186bcbe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f61b4a745f74e73ae77dfa285a9d849" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 42443 } } peers { permanent_uuid: "554a402446b245919090f648186bcbe2" member_type: VOTER last_known_addr { host: "127.13.99.61" port: 43587 } } peers { permanent_uuid: "bbcd4ccf923040a09d2dc68313713137" member_type: VOTER last_known_addr { host: "127.13.99.60" port: 38807 } }
I20260812 09:47:55.391616 3830 sys_catalog.cc:455] T 00000000000000000000000000000000 P 554a402446b245919090f648186bcbe2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 554a402446b245919090f648186bcbe2. Latest consensus state: current_term: 1 leader_uuid: "554a402446b245919090f648186bcbe2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f61b4a745f74e73ae77dfa285a9d849" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 42443 } } peers { permanent_uuid: "554a402446b245919090f648186bcbe2" member_type: VOTER last_known_addr { host: "127.13.99.61" port: 43587 } } peers { permanent_uuid: "bbcd4ccf923040a09d2dc68313713137" member_type: VOTER last_known_addr { host: "127.13.99.60" port: 38807 } } }
I20260812 09:47:55.391767 3830 sys_catalog.cc:458] T 00000000000000000000000000000000 P 554a402446b245919090f648186bcbe2 [sys.catalog]: This master's current role is: LEADER
I20260812 09:47:55.392697 3833 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 09:47:55.393669 3833 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 09:47:55.398660 3371 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4f61b4a745f74e73ae77dfa285a9d849 [term 1 FOLLOWER]: Refusing update from remote peer 554a402446b245919090f648186bcbe2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 09:47:55.398660 3666 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bbcd4ccf923040a09d2dc68313713137 [term 1 FOLLOWER]: Refusing update from remote peer 554a402446b245919090f648186bcbe2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 09:47:55.399351 3805 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 554a402446b245919090f648186bcbe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbcd4ccf923040a09d2dc68313713137" member_type: VOTER last_known_addr { host: "127.13.99.60" port: 38807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 09:47:55.399572 3830 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 554a402446b245919090f648186bcbe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f61b4a745f74e73ae77dfa285a9d849" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 42443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 09:47:55.403193 3782 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbcd4ccf923040a09d2dc68313713137 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 554a402446b245919090f648186bcbe2. Latest consensus state: current_term: 1 leader_uuid: "554a402446b245919090f648186bcbe2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f61b4a745f74e73ae77dfa285a9d849" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 42443 } } peers { permanent_uuid: "554a402446b245919090f648186bcbe2" member_type: VOTER last_known_addr { host: "127.13.99.61" port: 43587 } } peers { permanent_uuid: "bbcd4ccf923040a09d2dc68313713137" member_type: VOTER last_known_addr { host: "127.13.99.60" port: 38807 } } }
I20260812 09:47:55.403234 3761 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f61b4a745f74e73ae77dfa285a9d849 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 554a402446b245919090f648186bcbe2. Latest consensus state: current_term: 1 leader_uuid: "554a402446b245919090f648186bcbe2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f61b4a745f74e73ae77dfa285a9d849" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 42443 } } peers { permanent_uuid: "554a402446b245919090f648186bcbe2" member_type: VOTER last_known_addr { host: "127.13.99.61" port: 43587 } } peers { permanent_uuid: "bbcd4ccf923040a09d2dc68313713137" member_type: VOTER last_known_addr { host: "127.13.99.60" port: 38807 } } }
I20260812 09:47:55.403327 3782 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbcd4ccf923040a09d2dc68313713137 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 09:47:55.403342 3761 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f61b4a745f74e73ae77dfa285a9d849 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 09:47:55.405877 3835 mvcc.cc:205] Tried to move back new op lower bound from 7317618996825866240 to 7317618996797673472. Current Snapshot: MvccSnapshot[applied={T|T < 7317618996825866240}]
I20260812 09:47:55.405982 3805 sys_catalog.cc:455] T 00000000000000000000000000000000 P 554a402446b245919090f648186bcbe2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "554a402446b245919090f648186bcbe2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f61b4a745f74e73ae77dfa285a9d849" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 42443 } } peers { permanent_uuid: "554a402446b245919090f648186bcbe2" member_type: VOTER last_known_addr { host: "127.13.99.61" port: 43587 } } peers { permanent_uuid: "bbcd4ccf923040a09d2dc68313713137" member_type: VOTER last_known_addr { host: "127.13.99.60" port: 38807 } } }
I20260812 09:47:55.406111 3805 sys_catalog.cc:458] T 00000000000000000000000000000000 P 554a402446b245919090f648186bcbe2 [sys.catalog]: This master's current role is: LEADER
I20260812 09:47:55.406164 3761 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f61b4a745f74e73ae77dfa285a9d849 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "554a402446b245919090f648186bcbe2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f61b4a745f74e73ae77dfa285a9d849" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 42443 } } peers { permanent_uuid: "554a402446b245919090f648186bcbe2" member_type: VOTER last_known_addr { host: "127.13.99.61" port: 43587 } } peers { permanent_uuid: "bbcd4ccf923040a09d2dc68313713137" member_type: VOTER last_known_addr { host: "127.13.99.60" port: 38807 } } }
I20260812 09:47:55.406004 3830 sys_catalog.cc:455] T 00000000000000000000000000000000 P 554a402446b245919090f648186bcbe2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "554a402446b245919090f648186bcbe2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f61b4a745f74e73ae77dfa285a9d849" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 42443 } } peers { permanent_uuid: "554a402446b245919090f648186bcbe2" member_type: VOTER last_known_addr { host: "127.13.99.61" port: 43587 } } peers { permanent_uuid: "bbcd4ccf923040a09d2dc68313713137" member_type: VOTER last_known_addr { host: "127.13.99.60" port: 38807 } } }
I20260812 09:47:55.406270 3830 sys_catalog.cc:458] T 00000000000000000000000000000000 P 554a402446b245919090f648186bcbe2 [sys.catalog]: This master's current role is: LEADER
I20260812 09:47:55.406268 3761 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f61b4a745f74e73ae77dfa285a9d849 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 09:47:55.407217 3782 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbcd4ccf923040a09d2dc68313713137 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "554a402446b245919090f648186bcbe2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f61b4a745f74e73ae77dfa285a9d849" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 42443 } } peers { permanent_uuid: "554a402446b245919090f648186bcbe2" member_type: VOTER last_known_addr { host: "127.13.99.61" port: 43587 } } peers { permanent_uuid: "bbcd4ccf923040a09d2dc68313713137" member_type: VOTER last_known_addr { host: "127.13.99.60" port: 38807 } } }
I20260812 09:47:55.407330 3782 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbcd4ccf923040a09d2dc68313713137 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 09:47:55.407737 3833 catalog_manager.cc:1347] Generated new cluster ID: 3b931b35a8104ff48d23872867d0b759
I20260812 09:47:55.407792 3833 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 09:47:55.423189 3833 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 09:47:55.424265 3833 catalog_manager.cc:1504] Loading token signing keys...
I20260812 09:47:55.438616 3833 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 554a402446b245919090f648186bcbe2: Generated new TSK 0
I20260812 09:47:55.439292 3833 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 09:47:56.061583 3802 catalog_manager.cc:1259] Loaded cluster ID: 3b931b35a8104ff48d23872867d0b759
I20260812 09:47:56.061656 3802 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bbcd4ccf923040a09d2dc68313713137: loading cluster ID for follower catalog manager: success
I20260812 09:47:56.063455 3802 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bbcd4ccf923040a09d2dc68313713137: acquiring CA information for follower catalog manager: success
I20260812 09:47:56.064631 3802 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bbcd4ccf923040a09d2dc68313713137: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 09:47:56.068977 3780 catalog_manager.cc:1259] Loaded cluster ID: 3b931b35a8104ff48d23872867d0b759
I20260812 09:47:56.069063 3780 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4f61b4a745f74e73ae77dfa285a9d849: loading cluster ID for follower catalog manager: success
I20260812 09:47:56.070627 3780 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4f61b4a745f74e73ae77dfa285a9d849: acquiring CA information for follower catalog manager: success
I20260812 09:47:56.071731 3780 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4f61b4a745f74e73ae77dfa285a9d849: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 09:47:56.498006 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 3298
I20260812 09:47:56.514144 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 3435
I20260812 09:47:56.529335 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 3598
2026-08-12T09:47:56Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3638 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786527905902985-13708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 09:47:56 dist-test-slave-jq4t krb5kdc[3864](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 09:47:56 dist-test-slave-jq4t krb5kdc[3864](info): set up 2 sockets
Aug 12 09:47:56 dist-test-slave-jq4t krb5kdc[3864](info): commencing operation
krb5kdc: starting...
W20260812 09:47:57.817032 13708 mini_kdc.cc:121] Time spent starting KDC: real 1.261s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786527905902985-13708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786527905902985-13708-0/krb5kdc/test-user.keytab.
Aug 12 09:47:57 dist-test-slave-jq4t krb5kdc[3864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T09:47:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T09:47:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.99.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.99.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.99.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab.
Entry for principal kudu/127.13.99.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab.
I20260812 09:47:57.948163 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786527905902985-13708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786527905902985-13708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786527905902985-13708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.62:40077
--webserver_interface=127.13.99.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab
--principal=kudu/127.13.99.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:42339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.99.62:40077
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:47:58.062781 3889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:47:58.063042 3889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:47:58.063083 3889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:47:58.067097 3889 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 09:47:58.067149 3889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:47:58.067169 3889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:47:58.067185 3889 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 09:47:58.067245 3889 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 09:47:58.072415 3889 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:42339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786527905902985-13708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786527905902985-13708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.99.62:40077
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab
--principal=kudu/127.13.99.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.62:40077
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.99.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3889
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786527905902985-13708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:47:58.073474 3889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:47:58.074539 3889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:47:58.083274 3894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:47:58.083292 3896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:47:58.083376 3889 server_base.cc:1034] running on GCE node
W20260812 09:47:58.083258 3906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:47:58.084156 3889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:47:58.085300 3889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:47:58.086468 3889 hybrid_clock.cc:648] HybridClock initialized: now 1786528078086441 us; error 41 us; skew 500 ppm
Aug 12 09:47:58 dist-test-slave-jq4t krb5kdc[3864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528078, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:47:58.093492 3889 init.cc:377] Logged in from keytab as kudu/127.13.99.62@KRBTEST.COM (short username kudu)
I20260812 09:47:58.094502 3889 webserver.cc:460] Webserver started at http://127.13.99.62:37763/ using document root <none> and password file <none>
I20260812 09:47:58.095165 3889 fs_manager.cc:359] Metadata directory not provided
I20260812 09:47:58.095261 3889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:47:58.095504 3889 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:47:58.097672 3889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786527905902985-13708-0/minicluster-data/master-0/data/instance:
uuid: "ba5cce70570a4406b0dfe893d2262992"
format_stamp: "Formatted at 2026-08-12 09:47:58 on dist-test-slave-jq4t"
server_key: "0c37cae1e4d0b6389eb2297cfd9da8dd"
server_key_iv: "dfb673e38daae88d99ec0c016bc395fe"
server_key_version: "encryptionkey@0"
I20260812 09:47:58.098304 3889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786527905902985-13708-0/minicluster-data/master-0/wal/instance:
uuid: "ba5cce70570a4406b0dfe893d2262992"
format_stamp: "Formatted at 2026-08-12 09:47:58 on dist-test-slave-jq4t"
server_key: "0c37cae1e4d0b6389eb2297cfd9da8dd"
server_key_iv: "dfb673e38daae88d99ec0c016bc395fe"
server_key_version: "encryptionkey@0"
I20260812 09:47:58.102066 3889 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260812 09:47:58.104403 3915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:47:58.105680 3889 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260812 09:47:58.105825 3889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786527905902985-13708-0/minicluster-data/master-0/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786527905902985-13708-0/minicluster-data/master-0/wal
uuid: "ba5cce70570a4406b0dfe893d2262992"
format_stamp: "Formatted at 2026-08-12 09:47:58 on dist-test-slave-jq4t"
server_key: "0c37cae1e4d0b6389eb2297cfd9da8dd"
server_key_iv: "dfb673e38daae88d99ec0c016bc395fe"
server_key_version: "encryptionkey@0"
I20260812 09:47:58.105939 3889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786527905902985-13708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786527905902985-13708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786527905902985-13708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:47:58.121559 3889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:47:58.124967 3889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:47:58.125173 3889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:47:58.132050 3889 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.62:40077
I20260812 09:47:58.132154 4020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.62:40077 every 8 connection(s)
I20260812 09:47:58.133056 3889 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
I20260812 09:47:58.134636 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 3889
I20260812 09:47:58.134778 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786527905902985-13708-0/minicluster-data/master-0/wal/instance
I20260812 09:47:58.135062 13708 external_mini_cluster.cc:1428] Setting key 261de0cbcefa9c12b4980356d7b782f7
I20260812 09:47:58.136116 4021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:47:58.142936 4021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba5cce70570a4406b0dfe893d2262992: Bootstrap starting.
Aug 12 09:47:58 dist-test-slave-jq4t krb5kdc[3864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
I20260812 09:47:58.145478 4021 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba5cce70570a4406b0dfe893d2262992: Neither blocks nor log segments found. Creating new log.
I20260812 09:47:58.146448 4021 log.cc:824] T 00000000000000000000000000000000 P ba5cce70570a4406b0dfe893d2262992: Log is configured to *not* fsync() on all Append() calls
I20260812 09:47:58.148610 4021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba5cce70570a4406b0dfe893d2262992: No bootstrap required, opened a new log
I20260812 09:47:58.150161 4026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:47:58.136396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41354 (local address 127.13.99.62:40077)
0812 09:47:58.136891 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0812 09:47:58.136910 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0812 09:47:58.136951 (+ 41us) server_negotiation.cc:407] Connection header received
0812 09:47:58.137808 (+ 857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:47:58.137832 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:47:58.138226 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:47:58.138593 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:47:58.139250 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:47:58.140759 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:47:58.141577 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:47:58.142279 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:47:58.142438 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:47:58.145145 (+ 2707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:47:58.145150 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:47:58.145162 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:47:58.145188 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:47:58.147082 (+ 1894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:47:58.147519 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:47:58.147525 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:47:58.147531 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:47:58.147664 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:47:58.147919 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:47:58.147922 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:47:58.147924 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:47:58.148411 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:47:58.148652 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0812 09:47:58.149007 (+ 355us) server_negotiation.cc:299] Negotiation successful
0812 09:47:58.149289 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":141,"threads_started":1}
I20260812 09:47:58.153112 13708 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 09:47:58.152954 4021 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba5cce70570a4406b0dfe893d2262992 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba5cce70570a4406b0dfe893d2262992" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 40077 } }
I20260812 09:47:58.153198 4021 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba5cce70570a4406b0dfe893d2262992 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:47:58.153278 4021 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba5cce70570a4406b0dfe893d2262992 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba5cce70570a4406b0dfe893d2262992, State: Initialized, Role: FOLLOWER
I20260812 09:47:58.153707 4021 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba5cce70570a4406b0dfe893d2262992 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba5cce70570a4406b0dfe893d2262992" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 40077 } }
I20260812 09:47:58.153827 4021 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba5cce70570a4406b0dfe893d2262992 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 09:47:58.153908 4021 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba5cce70570a4406b0dfe893d2262992 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 09:47:58.154039 4021 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba5cce70570a4406b0dfe893d2262992 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:47:58.154875 4021 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba5cce70570a4406b0dfe893d2262992 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba5cce70570a4406b0dfe893d2262992" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 40077 } }
I20260812 09:47:58.155218 4021 leader_election.cc:302] T 00000000000000000000000000000000 P ba5cce70570a4406b0dfe893d2262992 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba5cce70570a4406b0dfe893d2262992; no voters:
I20260812 09:47:58.155519 4021 leader_election.cc:288] T 00000000000000000000000000000000 P ba5cce70570a4406b0dfe893d2262992 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 09:47:58.155689 4031 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba5cce70570a4406b0dfe893d2262992 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 09:47:58.156013 4031 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba5cce70570a4406b0dfe893d2262992 [term 1 LEADER]: Becoming Leader. State: Replica: ba5cce70570a4406b0dfe893d2262992, State: Running, Role: LEADER
I20260812 09:47:58.156344 4031 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba5cce70570a4406b0dfe893d2262992 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba5cce70570a4406b0dfe893d2262992" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 40077 } }
I20260812 09:47:58.156823 4021 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba5cce70570a4406b0dfe893d2262992 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 09:47:58.157984 4035 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba5cce70570a4406b0dfe893d2262992 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba5cce70570a4406b0dfe893d2262992. Latest consensus state: current_term: 1 leader_uuid: "ba5cce70570a4406b0dfe893d2262992" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba5cce70570a4406b0dfe893d2262992" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 40077 } } }
I20260812 09:47:58.158113 4035 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba5cce70570a4406b0dfe893d2262992 [sys.catalog]: This master's current role is: LEADER
I20260812 09:47:58.158496 4056 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 09:47:58.158460 4034 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba5cce70570a4406b0dfe893d2262992 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ba5cce70570a4406b0dfe893d2262992" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba5cce70570a4406b0dfe893d2262992" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 40077 } } }
I20260812 09:47:58.158635 4034 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba5cce70570a4406b0dfe893d2262992 [sys.catalog]: This master's current role is: LEADER
I20260812 09:47:58.161535 4056 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 09:47:58.167116 4056 catalog_manager.cc:1347] Generated new cluster ID: a31b5ba8b23945d2aeb3339c8e3fba20
I20260812 09:47:58.167193 4056 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 09:47:58.193176 4056 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 09:47:58.194170 4056 catalog_manager.cc:1504] Loading token signing keys...
I20260812 09:47:58.199230 4056 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ba5cce70570a4406b0dfe893d2262992: Generated new TSK 0
I20260812 09:47:58.199888 4056 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 09:47:58.208966 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 3889
2026-08-12T09:47:58Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1676 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 09:47:58 dist-test-slave-jq4t krb5kdc[4086](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 09:47:58 dist-test-slave-jq4t krb5kdc[4086](info): set up 2 sockets
Aug 12 09:47:58 dist-test-slave-jq4t krb5kdc[4086](info): commencing operation
krb5kdc: starting...
W20260812 09:47:59.483618 13708 mini_kdc.cc:121] Time spent starting KDC: real 1.251s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/krb5kdc/test-user.keytab.
Aug 12 09:47:59 dist-test-slave-jq4t krb5kdc[4086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528079, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T09:47:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T09:47:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.99.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.99.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.99.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab.
Entry for principal kudu/127.13.99.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab.
I20260812 09:47:59.614096 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.62:42637
--webserver_interface=127.13.99.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab
--principal=kudu/127.13.99.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:33335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.99.62:42637
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:47:59.730650 4116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:47:59.731030 4116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:47:59.731144 4116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:47:59.735394 4116 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 09:47:59.735491 4116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:47:59.735533 4116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:47:59.735597 4116 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 09:47:59.735636 4116 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 09:47:59.741094 4116 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:33335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.99.62:42637
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab
--principal=kudu/127.13.99.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.62:42637
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.99.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4116
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:47:59.742244 4116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:47:59.743310 4116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:47:59.751945 4129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:47:59.751948 4122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:47:59.751947 4125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:47:59.752359 4116 server_base.cc:1034] running on GCE node
I20260812 09:47:59.753007 4116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:47:59.754132 4116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:47:59.755311 4116 hybrid_clock.cc:648] HybridClock initialized: now 1786528079755277 us; error 48 us; skew 500 ppm
Aug 12 09:47:59 dist-test-slave-jq4t krb5kdc[4086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528079, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:47:59.758272 4116 init.cc:377] Logged in from keytab as kudu/127.13.99.62@KRBTEST.COM (short username kudu)
I20260812 09:47:59.759168 4116 webserver.cc:460] Webserver started at http://127.13.99.62:40349/ using document root <none> and password file <none>
I20260812 09:47:59.759749 4116 fs_manager.cc:359] Metadata directory not provided
I20260812 09:47:59.759842 4116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:47:59.760087 4116 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:47:59.762199 4116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/master-0/data/instance:
uuid: "83d811857241469db629144445c55d66"
format_stamp: "Formatted at 2026-08-12 09:47:59 on dist-test-slave-jq4t"
server_key: "fdf1de0a3059815d8e6a12d9b4a6761a"
server_key_iv: "78086bf54cfa847ca62731e158cbefed"
server_key_version: "encryptionkey@0"
I20260812 09:47:59.762781 4116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/master-0/wal/instance:
uuid: "83d811857241469db629144445c55d66"
format_stamp: "Formatted at 2026-08-12 09:47:59 on dist-test-slave-jq4t"
server_key: "fdf1de0a3059815d8e6a12d9b4a6761a"
server_key_iv: "78086bf54cfa847ca62731e158cbefed"
server_key_version: "encryptionkey@0"
I20260812 09:47:59.766552 4116 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260812 09:47:59.769009 4138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:47:59.770282 4116 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 09:47:59.770426 4116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/master-0/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/master-0/wal
uuid: "83d811857241469db629144445c55d66"
format_stamp: "Formatted at 2026-08-12 09:47:59 on dist-test-slave-jq4t"
server_key: "fdf1de0a3059815d8e6a12d9b4a6761a"
server_key_iv: "78086bf54cfa847ca62731e158cbefed"
server_key_version: "encryptionkey@0"
I20260812 09:47:59.770547 4116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:47:59.797565 4116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:47:59.801208 4116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:47:59.801426 4116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:47:59.809020 4116 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.62:42637
I20260812 09:47:59.809041 4237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.62:42637 every 8 connection(s)
I20260812 09:47:59.810009 4116 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
I20260812 09:47:59.811827 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 4116
I20260812 09:47:59.811966 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/master-0/wal/instance
I20260812 09:47:59.812295 13708 external_mini_cluster.cc:1428] Setting key d7dbf4201a73ab77a44038f39e8c5c30
I20260812 09:47:59.813205 4239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:47:59.819015 4239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83d811857241469db629144445c55d66: Bootstrap starting.
Aug 12 09:47:59 dist-test-slave-jq4t krb5kdc[4086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528079, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
I20260812 09:47:59.821517 4239 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 83d811857241469db629144445c55d66: Neither blocks nor log segments found. Creating new log.
I20260812 09:47:59.822247 4239 log.cc:824] T 00000000000000000000000000000000 P 83d811857241469db629144445c55d66: Log is configured to *not* fsync() on all Append() calls
I20260812 09:47:59.824098 4239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83d811857241469db629144445c55d66: No bootstrap required, opened a new log
I20260812 09:47:59.826401 4245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:47:59.813800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49470 (local address 127.13.99.62:42637)
0812 09:47:59.814279 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0812 09:47:59.814295 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 09:47:59.814331 (+ 36us) server_negotiation.cc:407] Connection header received
0812 09:47:59.815003 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:47:59.815021 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:47:59.815350 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:47:59.815708 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:47:59.816559 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:47:59.817565 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:47:59.818430 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:47:59.818906 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:47:59.819087 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:47:59.821857 (+ 2770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:47:59.821862 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:47:59.821876 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:47:59.821902 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:47:59.823605 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:47:59.824066 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:47:59.824071 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:47:59.824076 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:47:59.824185 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:47:59.824505 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:47:59.824511 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:47:59.824513 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:47:59.824901 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:47:59.825118 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0812 09:47:59.825421 (+ 303us) server_negotiation.cc:299] Negotiation successful
0812 09:47:59.825710 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":162,"threads_started":1}
I20260812 09:47:59.827795 4239 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83d811857241469db629144445c55d66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83d811857241469db629144445c55d66" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 42637 } }
I20260812 09:47:59.828043 4239 raft_consensus.cc:374] T 00000000000000000000000000000000 P 83d811857241469db629144445c55d66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:47:59.828127 4239 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83d811857241469db629144445c55d66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83d811857241469db629144445c55d66, State: Initialized, Role: FOLLOWER
I20260812 09:47:59.828657 4239 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83d811857241469db629144445c55d66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83d811857241469db629144445c55d66" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 42637 } }
I20260812 09:47:59.828814 4239 raft_consensus.cc:388] T 00000000000000000000000000000000 P 83d811857241469db629144445c55d66 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 09:47:59.828887 4239 raft_consensus.cc:482] T 00000000000000000000000000000000 P 83d811857241469db629144445c55d66 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 09:47:59.828987 4239 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83d811857241469db629144445c55d66 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:47:59.829881 4239 raft_consensus.cc:504] T 00000000000000000000000000000000 P 83d811857241469db629144445c55d66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83d811857241469db629144445c55d66" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 42637 } }
I20260812 09:47:59.830227 4239 leader_election.cc:302] T 00000000000000000000000000000000 P 83d811857241469db629144445c55d66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 83d811857241469db629144445c55d66; no voters:
I20260812 09:47:59.830581 4239 leader_election.cc:288] T 00000000000000000000000000000000 P 83d811857241469db629144445c55d66 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 09:47:59.830657 4251 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 83d811857241469db629144445c55d66 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 09:47:59.830941 4251 raft_consensus.cc:686] T 00000000000000000000000000000000 P 83d811857241469db629144445c55d66 [term 1 LEADER]: Becoming Leader. State: Replica: 83d811857241469db629144445c55d66, State: Running, Role: LEADER
I20260812 09:47:59.831274 4251 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83d811857241469db629144445c55d66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83d811857241469db629144445c55d66" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 42637 } }
I20260812 09:47:59.831811 4239 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83d811857241469db629144445c55d66 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 09:47:59.833070 4254 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83d811857241469db629144445c55d66 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83d811857241469db629144445c55d66. Latest consensus state: current_term: 1 leader_uuid: "83d811857241469db629144445c55d66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83d811857241469db629144445c55d66" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 42637 } } }
I20260812 09:47:59.833026 4253 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83d811857241469db629144445c55d66 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "83d811857241469db629144445c55d66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83d811857241469db629144445c55d66" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 42637 } } }
I20260812 09:47:59.833177 4253 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83d811857241469db629144445c55d66 [sys.catalog]: This master's current role is: LEADER
I20260812 09:47:59.833177 4254 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83d811857241469db629144445c55d66 [sys.catalog]: This master's current role is: LEADER
I20260812 09:47:59.833642 4270 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 09:47:59.836764 4270 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 09:47:59.843099 4270 catalog_manager.cc:1347] Generated new cluster ID: b991061c7a9e413fa211af0ca6b9e99c
I20260812 09:47:59.843171 4270 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 09:47:59.852011 4270 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 09:47:59.853241 4270 catalog_manager.cc:1504] Loading token signing keys...
I20260812 09:47:59.864662 4270 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 83d811857241469db629144445c55d66: Generated new TSK 0
I20260812 09:47:59.865535 4270 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.99.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.99.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.99.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.1.keytab.
Entry for principal kudu/127.13.99.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.1.keytab.
I20260812 09:47:59.903919 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.99.1:0
--local_ip_for_outbound_sockets=127.13.99.1
--webserver_interface=127.13.99.1
--webserver_port=0
--tserver_master_addrs=127.13.99.62:42637
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.1.keytab
--principal=kudu/127.13.99.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:33335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:48:00.021322 4297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:48:00.021580 4297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:48:00.021622 4297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:48:00.025660 4297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:48:00.025717 4297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:48:00.025812 4297 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.99.1
I20260812 09:48:00.031081 4297 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:33335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.1.keytab
--principal=kudu/127.13.99.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.99.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.99.1
--webserver_port=0
--tserver_master_addrs=127.13.99.62:42637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4297
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.99.1
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:48:00.032291 4297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:48:00.033502 4297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:48:00.043073 4310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:48:00.043076 4308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:48:00.043107 4303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:48:00.043358 4297 server_base.cc:1034] running on GCE node
I20260812 09:48:00.043812 4297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:48:00.044435 4297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:48:00.045674 4297 hybrid_clock.cc:648] HybridClock initialized: now 1786528080045660 us; error 28 us; skew 500 ppm
Aug 12 09:48:00 dist-test-slave-jq4t krb5kdc[4086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528080, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:48:00.048470 4297 init.cc:377] Logged in from keytab as kudu/127.13.99.1@KRBTEST.COM (short username kudu)
I20260812 09:48:00.049391 4297 webserver.cc:460] Webserver started at http://127.13.99.1:37531/ using document root <none> and password file <none>
I20260812 09:48:00.049970 4297 fs_manager.cc:359] Metadata directory not provided
I20260812 09:48:00.050072 4297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:48:00.050308 4297 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:48:00.052356 4297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-0/data/instance:
uuid: "a808eee3e1c14f9488cefe0d16ccc66c"
format_stamp: "Formatted at 2026-08-12 09:48:00 on dist-test-slave-jq4t"
server_key: "cc68ccaa2e555cece62a47eb75f4b47c"
server_key_iv: "5419744aa6331d0fba4b0e14ae3946db"
server_key_version: "encryptionkey@0"
I20260812 09:48:00.053011 4297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-0/wal/instance:
uuid: "a808eee3e1c14f9488cefe0d16ccc66c"
format_stamp: "Formatted at 2026-08-12 09:48:00 on dist-test-slave-jq4t"
server_key: "cc68ccaa2e555cece62a47eb75f4b47c"
server_key_iv: "5419744aa6331d0fba4b0e14ae3946db"
server_key_version: "encryptionkey@0"
I20260812 09:48:00.056690 4297 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260812 09:48:00.059005 4327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:48:00.060237 4297 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 09:48:00.060374 4297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-0/wal
uuid: "a808eee3e1c14f9488cefe0d16ccc66c"
format_stamp: "Formatted at 2026-08-12 09:48:00 on dist-test-slave-jq4t"
server_key: "cc68ccaa2e555cece62a47eb75f4b47c"
server_key_iv: "5419744aa6331d0fba4b0e14ae3946db"
server_key_version: "encryptionkey@0"
I20260812 09:48:00.060551 4297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:48:00.086046 4297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:48:00.089026 4297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:48:00.089184 4297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:48:00.089725 4297 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 09:48:00.090502 4297 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:48:00.090559 4297 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 09:48:00.090601 4297 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:48:00.090622 4297 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 09:48:00.099490 4297 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.1:43137
I20260812 09:48:00.099524 4536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.1:43137 every 8 connection(s)
I20260812 09:48:00.100387 4297 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-0/data/info.pb
I20260812 09:48:00.102123 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 4297
I20260812 09:48:00.102288 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-0/wal/instance
I20260812 09:48:00.102614 13708 external_mini_cluster.cc:1428] Setting key e642e680047f76c6cc006dc15fde9e56
Aug 12 09:48:00 dist-test-slave-jq4t krb5kdc[4086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528080, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.1@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.99.2@KRBTEST.COM; defaulting to no policy
I20260812 09:48:00.116252 4245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:00.102495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.1:39667 (local address 127.13.99.62:42637)
0812 09:48:00.102645 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0812 09:48:00.102667 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0812 09:48:00.103577 (+ 910us) server_negotiation.cc:407] Connection header received
0812 09:48:00.105437 (+ 1860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:48:00.105440 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:48:00.105494 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:48:00.105564 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:48:00.107115 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:48:00.107996 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:48:00.109265 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:48:00.109740 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:48:00.109871 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:48:00.112387 (+ 2516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:48:00.112393 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:48:00.112396 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:48:00.112431 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:48:00.114215 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:48:00.114695 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:48:00.114698 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:48:00.114699 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:48:00.114776 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:48:00.115124 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:48:00.115128 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:48:00.115130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:48:00.115326 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:48:00.115439 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0812 09:48:00.115936 (+ 497us) server_negotiation.cc:299] Negotiation successful
0812 09:48:00.116072 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 09:48:00.117094 4540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:00.102872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:42637 (local address 127.13.99.1:39667)
0812 09:48:00.103417 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0812 09:48:00.103471 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0812 09:48:00.105178 (+ 1707us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:48:00.105771 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:48:00.105783 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:48:00.106320 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:48:00.106949 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:48:00.106963 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:48:00.108137 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:48:00.108140 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:48:00.109131 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:48:00.109142 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:48:00.109922 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:48:00.109925 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:48:00.110093 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:48:00.110725 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:48:00.110742 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:48:00.112200 (+ 1458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:48:00.114330 (+ 2130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:48:00.114335 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:48:00.114347 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:48:00.114599 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:48:00.114863 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:48:00.114866 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:48:00.114867 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:48:00.115016 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:48:00.115441 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:48:00.115448 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:48:00.115698 (+ 250us) client_negotiation.cc:769] Sending connection context
0812 09:48:00.115891 (+ 193us) client_negotiation.cc:240] Negotiation successful
0812 09:48:00.116218 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":167,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.99.2@KRBTEST.COM" created.
I20260812 09:48:00.118409 4537 heartbeater.cc:347] Connected to a master server at 127.13.99.62:42637
I20260812 09:48:00.118695 4537 heartbeater.cc:464] Registering TS with master...
I20260812 09:48:00.119325 4537 heartbeater.cc:511] Master 127.13.99.62:42637 requested a full tablet report, sending...
I20260812 09:48:00.121052 4166 ts_manager.cc:194] Registered new tserver with Master: a808eee3e1c14f9488cefe0d16ccc66c (127.13.99.1:43137)
I20260812 09:48:00.122200 4166 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.99.1@KRBTEST.COM'} at 127.13.99.1:39667
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.99.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.2.keytab.
Entry for principal kudu/127.13.99.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.2.keytab.
I20260812 09:48:00.133903 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.99.2:0
--local_ip_for_outbound_sockets=127.13.99.2
--webserver_interface=127.13.99.2
--webserver_port=0
--tserver_master_addrs=127.13.99.62:42637
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.2.keytab
--principal=kudu/127.13.99.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:33335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:48:00.249238 4544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:48:00.249480 4544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:48:00.249519 4544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:48:00.253597 4544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:48:00.253652 4544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:48:00.253708 4544 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.99.2
I20260812 09:48:00.258745 4544 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:33335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.2.keytab
--principal=kudu/127.13.99.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.99.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.99.2
--webserver_port=0
--tserver_master_addrs=127.13.99.62:42637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4544
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.99.2
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:48:00.259668 4544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:48:00.260746 4544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:48:00.271277 4557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:48:00.271293 4560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:48:00.271450 4544 server_base.cc:1034] running on GCE node
W20260812 09:48:00.271600 4554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:48:00.272038 4544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:48:00.272714 4544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:48:00.273921 4544 hybrid_clock.cc:648] HybridClock initialized: now 1786528080273900 us; error 29 us; skew 500 ppm
Aug 12 09:48:00 dist-test-slave-jq4t krb5kdc[4086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528080, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:48:00.276758 4544 init.cc:377] Logged in from keytab as kudu/127.13.99.2@KRBTEST.COM (short username kudu)
I20260812 09:48:00.277609 4544 webserver.cc:460] Webserver started at http://127.13.99.2:35313/ using document root <none> and password file <none>
I20260812 09:48:00.278187 4544 fs_manager.cc:359] Metadata directory not provided
I20260812 09:48:00.278280 4544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:48:00.278517 4544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:48:00.280615 4544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-1/data/instance:
uuid: "0b12969c05f24d038e309045b035f69e"
format_stamp: "Formatted at 2026-08-12 09:48:00 on dist-test-slave-jq4t"
server_key: "4ae9a995fc90c118a5b17e256f83c06f"
server_key_iv: "6c459649e7869d8cb1627147f978999d"
server_key_version: "encryptionkey@0"
I20260812 09:48:00.281200 4544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-1/wal/instance:
uuid: "0b12969c05f24d038e309045b035f69e"
format_stamp: "Formatted at 2026-08-12 09:48:00 on dist-test-slave-jq4t"
server_key: "4ae9a995fc90c118a5b17e256f83c06f"
server_key_iv: "6c459649e7869d8cb1627147f978999d"
server_key_version: "encryptionkey@0"
I20260812 09:48:00.284915 4544 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260812 09:48:00.287303 4572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:48:00.288425 4544 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 09:48:00.288596 4544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-1/wal
uuid: "0b12969c05f24d038e309045b035f69e"
format_stamp: "Formatted at 2026-08-12 09:48:00 on dist-test-slave-jq4t"
server_key: "4ae9a995fc90c118a5b17e256f83c06f"
server_key_iv: "6c459649e7869d8cb1627147f978999d"
server_key_version: "encryptionkey@0"
I20260812 09:48:00.288710 4544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:48:00.315855 4544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:48:00.319067 4544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:48:00.319283 4544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:48:00.319912 4544 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 09:48:00.320807 4544 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:48:00.320881 4544 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 09:48:00.320955 4544 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:48:00.321009 4544 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 09:48:00.330322 4544 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.2:39701
I20260812 09:48:00.330353 4789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.2:39701 every 8 connection(s)
I20260812 09:48:00.331275 4544 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-1/data/info.pb
I20260812 09:48:00.331427 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 4544
I20260812 09:48:00.331542 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-1/wal/instance
I20260812 09:48:00.331890 13708 external_mini_cluster.cc:1428] Setting key 60c383bfd6baeb328f9b540f45a9ea45
Aug 12 09:48:00 dist-test-slave-jq4t krb5kdc[4086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528080, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.2@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.99.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.99.3@KRBTEST.COM" created.
I20260812 09:48:00.347873 4245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:00.333591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.2:41895 (local address 127.13.99.62:42637)
0812 09:48:00.333729 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0812 09:48:00.333735 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 09:48:00.334737 (+ 1002us) server_negotiation.cc:407] Connection header received
0812 09:48:00.336761 (+ 2024us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:48:00.336764 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:48:00.336822 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:48:00.336902 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:48:00.338109 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:48:00.338681 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:48:00.340421 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:48:00.340718 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:48:00.340808 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:48:00.343830 (+ 3022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:48:00.343834 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:48:00.343836 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:48:00.343863 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:48:00.345320 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:48:00.345936 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:48:00.345939 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:48:00.345941 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:48:00.346074 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:48:00.346507 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:48:00.346510 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:48:00.346512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:48:00.346798 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:48:00.346929 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0812 09:48:00.347415 (+ 486us) server_negotiation.cc:299] Negotiation successful
0812 09:48:00.347576 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260812 09:48:00.348439 4793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:00.334018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:42637 (local address 127.13.99.2:41895)
0812 09:48:00.334549 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0812 09:48:00.334615 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0812 09:48:00.336525 (+ 1910us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:48:00.337087 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:48:00.337095 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:48:00.337474 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:48:00.337947 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:48:00.337961 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:48:00.338818 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:48:00.338822 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:48:00.340282 (+ 1460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:48:00.340297 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:48:00.340885 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:48:00.340889 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:48:00.341131 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:48:00.341771 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:48:00.341797 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:48:00.343634 (+ 1837us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:48:00.345451 (+ 1817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:48:00.345457 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:48:00.345473 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:48:00.345791 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:48:00.346199 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:48:00.346206 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:48:00.346211 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:48:00.346408 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:48:00.346922 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:48:00.346927 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:48:00.347168 (+ 241us) client_negotiation.cc:769] Sending connection context
0812 09:48:00.347328 (+ 160us) client_negotiation.cc:240] Negotiation successful
0812 09:48:00.347577 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":121,"threads_started":1}
I20260812 09:48:00.349790 4791 heartbeater.cc:347] Connected to a master server at 127.13.99.62:42637
I20260812 09:48:00.350081 4791 heartbeater.cc:464] Registering TS with master...
I20260812 09:48:00.350713 4791 heartbeater.cc:511] Master 127.13.99.62:42637 requested a full tablet report, sending...
I20260812 09:48:00.352027 4166 ts_manager.cc:194] Registered new tserver with Master: 0b12969c05f24d038e309045b035f69e (127.13.99.2:39701)
I20260812 09:48:00.352959 4166 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.99.2@KRBTEST.COM'} at 127.13.99.2:41895
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.99.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.3.keytab.
Entry for principal kudu/127.13.99.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.3.keytab.
I20260812 09:48:00.363446 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.99.3:0
--local_ip_for_outbound_sockets=127.13.99.3
--webserver_interface=127.13.99.3
--webserver_port=0
--tserver_master_addrs=127.13.99.62:42637
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.3.keytab
--principal=kudu/127.13.99.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:33335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:48:00.481383 4797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:48:00.481630 4797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:48:00.481670 4797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:48:00.485550 4797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:48:00.485602 4797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:48:00.485658 4797 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.99.3
I20260812 09:48:00.490638 4797 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:33335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.3.keytab
--principal=kudu/127.13.99.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.99.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.99.3
--webserver_port=0
--tserver_master_addrs=127.13.99.62:42637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4797
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.99.3
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:48:00.491627 4797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:48:00.492729 4797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:48:00.502588 4810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:48:00.502691 4797 server_base.cc:1034] running on GCE node
W20260812 09:48:00.502619 4807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:48:00.502619 4813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:48:00.503212 4797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:48:00.503840 4797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:48:00.505060 4797 hybrid_clock.cc:648] HybridClock initialized: now 1786528080505030 us; error 48 us; skew 500 ppm
Aug 12 09:48:00 dist-test-slave-jq4t krb5kdc[4086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528080, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:48:00.507779 4797 init.cc:377] Logged in from keytab as kudu/127.13.99.3@KRBTEST.COM (short username kudu)
I20260812 09:48:00.508716 4797 webserver.cc:460] Webserver started at http://127.13.99.3:43579/ using document root <none> and password file <none>
I20260812 09:48:00.509268 4797 fs_manager.cc:359] Metadata directory not provided
I20260812 09:48:00.509359 4797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:48:00.509583 4797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:48:00.511548 4797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-2/data/instance:
uuid: "5cdb8b929df74926b65c64638cbbd3f5"
format_stamp: "Formatted at 2026-08-12 09:48:00 on dist-test-slave-jq4t"
server_key: "db9007ea05f29349965f91af0a28b62e"
server_key_iv: "aada7159e0a0a68d425fc6216d560d35"
server_key_version: "encryptionkey@0"
I20260812 09:48:00.512116 4797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-2/wal/instance:
uuid: "5cdb8b929df74926b65c64638cbbd3f5"
format_stamp: "Formatted at 2026-08-12 09:48:00 on dist-test-slave-jq4t"
server_key: "db9007ea05f29349965f91af0a28b62e"
server_key_iv: "aada7159e0a0a68d425fc6216d560d35"
server_key_version: "encryptionkey@0"
I20260812 09:48:00.515707 4797 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260812 09:48:00.518196 4823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:48:00.519512 4797 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260812 09:48:00.519673 4797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-2/wal
uuid: "5cdb8b929df74926b65c64638cbbd3f5"
format_stamp: "Formatted at 2026-08-12 09:48:00 on dist-test-slave-jq4t"
server_key: "db9007ea05f29349965f91af0a28b62e"
server_key_iv: "aada7159e0a0a68d425fc6216d560d35"
server_key_version: "encryptionkey@0"
I20260812 09:48:00.519794 4797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:48:00.545600 4797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:48:00.548915 4797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:48:00.549151 4797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:48:00.549778 4797 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 09:48:00.550578 4797 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:48:00.550628 4797 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 09:48:00.550668 4797 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:48:00.550711 4797 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 09:48:00.560297 4797 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.3:32841
I20260812 09:48:00.560355 5087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.3:32841 every 8 connection(s)
I20260812 09:48:00.561321 4797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-2/data/info.pb
Aug 12 09:48:00 dist-test-slave-jq4t krb5kdc[4086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528080, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.3@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
I20260812 09:48:00.571277 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 4797
I20260812 09:48:00.571400 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-2/wal/instance
I20260812 09:48:00.571691 13708 external_mini_cluster.cc:1428] Setting key f1ba2dc02fd8b963bc75bb8520029c04
I20260812 09:48:00.575428 4245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:00.563088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.3:54903 (local address 127.13.99.62:42637)
0812 09:48:00.563232 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0812 09:48:00.563237 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:48:00.564115 (+ 878us) server_negotiation.cc:407] Connection header received
0812 09:48:00.565617 (+ 1502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:48:00.565620 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:48:00.565668 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:48:00.565744 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:48:00.567054 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:48:00.567595 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:48:00.568827 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:48:00.569071 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:48:00.569164 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:48:00.571708 (+ 2544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:48:00.571711 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:48:00.571713 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:48:00.571738 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:48:00.573279 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:48:00.573833 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:48:00.573837 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:48:00.573839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:48:00.573925 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:48:00.574332 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:48:00.574335 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:48:00.574337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:48:00.574532 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:48:00.574642 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0812 09:48:00.575143 (+ 501us) server_negotiation.cc:299] Negotiation successful
0812 09:48:00.575270 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260812 09:48:00.576239 5092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:00.563488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:42637 (local address 127.13.99.3:54903)
0812 09:48:00.563968 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0812 09:48:00.564009 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0812 09:48:00.565420 (+ 1411us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:48:00.566024 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:48:00.566033 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:48:00.566408 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:48:00.566908 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:48:00.566921 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:48:00.567715 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:48:00.567718 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:48:00.568710 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:48:00.568721 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:48:00.569208 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:48:00.569210 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:48:00.569346 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:48:00.569986 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:48:00.570004 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:48:00.571515 (+ 1511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:48:00.573422 (+ 1907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:48:00.573428 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:48:00.573443 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:48:00.573719 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:48:00.574024 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:48:00.574028 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:48:00.574030 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:48:00.574227 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:48:00.574643 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:48:00.574650 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:48:00.574905 (+ 255us) client_negotiation.cc:769] Sending connection context
0812 09:48:00.575097 (+ 192us) client_negotiation.cc:240] Negotiation successful
0812 09:48:00.575429 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":119,"threads_started":1}
I20260812 09:48:00.577358 5088 heartbeater.cc:347] Connected to a master server at 127.13.99.62:42637
I20260812 09:48:00.577587 5088 heartbeater.cc:464] Registering TS with master...
I20260812 09:48:00.578034 5088 heartbeater.cc:511] Master 127.13.99.62:42637 requested a full tablet report, sending...
I20260812 09:48:00.578999 4166 ts_manager.cc:194] Registered new tserver with Master: 5cdb8b929df74926b65c64638cbbd3f5 (127.13.99.3:32841)
I20260812 09:48:00.579576 4166 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.99.3@KRBTEST.COM'} at 127.13.99.3:54903
I20260812 09:48:00.585994 13708 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 09:48:00.596673 4245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:00.588136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49476 (local address 127.13.99.62:42637)
0812 09:48:00.588313 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0812 09:48:00.588320 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 09:48:00.588553 (+ 233us) server_negotiation.cc:407] Connection header received
0812 09:48:00.588735 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:48:00.588739 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:48:00.588793 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:48:00.588883 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:48:00.589690 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:48:00.590262 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:48:00.591239 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:48:00.591501 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:48:00.591625 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:48:00.593330 (+ 1705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:48:00.593334 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:48:00.593337 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:48:00.593373 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:48:00.594707 (+ 1334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:48:00.595175 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:48:00.595179 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:48:00.595181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:48:00.595275 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:48:00.595597 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:48:00.595602 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:48:00.595604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:48:00.595864 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:48:00.596066 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0812 09:48:00.596341 (+ 275us) server_negotiation.cc:299] Negotiation successful
0812 09:48:00.596465 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260812 09:48:00.599503 4166 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49476:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 09:48:00.602900 4166 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 09:48:00.617694 5109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:00.611917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.3:32841 (local address 127.0.0.1:34668)
0812 09:48:00.612628 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0812 09:48:00.612680 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0812 09:48:00.612809 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:48:00.613545 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:48:00.613551 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:48:00.613580 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:48:00.613740 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:48:00.613747 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:48:00.614953 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:48:00.614960 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:48:00.616165 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:48:00.616175 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:48:00.617067 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:48:00.617071 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:48:00.617234 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:48:00.617292 (+ 58us) client_negotiation.cc:769] Sending connection context
0812 09:48:00.617413 (+ 121us) client_negotiation.cc:240] Negotiation successful
0812 09:48:00.617537 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"mutex_wait_us":46,"thread_start_us":81,"threads_started":1}
I20260812 09:48:00.617703 5119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:00.612110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34668 (local address 127.13.99.3:32841)
0812 09:48:00.613029 (+ 919us) server_negotiation.cc:206] Beginning negotiation
0812 09:48:00.613043 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 09:48:00.613069 (+ 26us) server_negotiation.cc:407] Connection header received
0812 09:48:00.613154 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:48:00.613160 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:48:00.613379 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:48:00.613514 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:48:00.613868 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:48:00.614812 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:48:00.616287 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:48:00.616936 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:48:00.617065 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:48:00.617229 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0812 09:48:00.617447 (+ 218us) server_negotiation.cc:299] Negotiation successful
0812 09:48:00.617537 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":784,"thread_start_us":144,"threads_started":1}
I20260812 09:48:00.618656 5114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:00.612104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.2:39701 (local address 127.0.0.1:59154)
0812 09:48:00.613044 (+ 940us) negotiation.cc:107] Waiting for socket to connect
0812 09:48:00.613060 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 09:48:00.613139 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:48:00.614278 (+ 1139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:48:00.614281 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:48:00.614304 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:48:00.614422 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:48:00.614433 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:48:00.615934 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:48:00.615937 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:48:00.617104 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:48:00.617114 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:48:00.618249 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:48:00.618253 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:48:00.618357 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:48:00.618375 (+ 18us) client_negotiation.cc:769] Sending connection context
0812 09:48:00.618418 (+ 43us) client_negotiation.cc:240] Negotiation successful
0812 09:48:00.618468 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":876,"mutex_wait_us":3,"thread_start_us":125,"threads_started":1}
I20260812 09:48:00.618903 5122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:00.612654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.1:43137 (local address 127.0.0.1:45880)
0812 09:48:00.613553 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0812 09:48:00.613566 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 09:48:00.613654 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:48:00.614148 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:48:00.614152 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:48:00.614172 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:48:00.614265 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:48:00.614271 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:48:00.615691 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:48:00.615694 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:48:00.617006 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:48:00.617018 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:48:00.618322 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:48:00.618326 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:48:00.618425 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:48:00.618450 (+ 25us) client_negotiation.cc:769] Sending connection context
0812 09:48:00.618684 (+ 234us) client_negotiation.cc:240] Negotiation successful
0812 09:48:00.618763 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":836,"mutex_wait_us":30,"thread_start_us":78,"threads_started":1}
I20260812 09:48:00.619288 5123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:00.613064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45880 (local address 127.13.99.1:43137)
0812 09:48:00.613397 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0812 09:48:00.613411 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 09:48:00.613679 (+ 268us) server_negotiation.cc:407] Connection header received
0812 09:48:00.613770 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:48:00.613774 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:48:00.613995 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:48:00.614149 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:48:00.614388 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:48:00.615530 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:48:00.617159 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:48:00.617989 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:48:00.618098 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:48:00.618259 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0812 09:48:00.619061 (+ 802us) server_negotiation.cc:299] Negotiation successful
0812 09:48:00.619156 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":118,"threads_started":1}
I20260812 09:48:00.619350 5120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:00.612544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59154 (local address 127.13.99.2:39701)
0812 09:48:00.613827 (+ 1283us) server_negotiation.cc:206] Beginning negotiation
0812 09:48:00.613838 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 09:48:00.613857 (+ 19us) server_negotiation.cc:407] Connection header received
0812 09:48:00.613923 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:48:00.613928 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:48:00.614120 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:48:00.614584 (+ 464us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:48:00.614743 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:48:00.615790 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:48:00.617244 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:48:00.618114 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:48:00.618852 (+ 738us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:48:00.619012 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0812 09:48:00.619065 (+ 53us) server_negotiation.cc:299] Negotiation successful
0812 09:48:00.619159 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1185,"thread_start_us":106,"threads_started":1}
I20260812 09:48:00.621294 4929 tablet_service.cc:1467] Processing CreateTablet for tablet 2b15723ad5254743b1195fce5a796a52 (DEFAULT_TABLE table=test-table [id=f562716082a34e278473bf195c2fa19f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 09:48:00.622040 4648 tablet_service.cc:1467] Processing CreateTablet for tablet 2b15723ad5254743b1195fce5a796a52 (DEFAULT_TABLE table=test-table [id=f562716082a34e278473bf195c2fa19f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 09:48:00.622503 4929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b15723ad5254743b1195fce5a796a52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:48:00.622138 4408 tablet_service.cc:1467] Processing CreateTablet for tablet 2b15723ad5254743b1195fce5a796a52 (DEFAULT_TABLE table=test-table [id=f562716082a34e278473bf195c2fa19f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 09:48:00.622835 4648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b15723ad5254743b1195fce5a796a52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:48:00.623095 4408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b15723ad5254743b1195fce5a796a52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:48:00.629184 5125 tablet_bootstrap.cc:492] T 2b15723ad5254743b1195fce5a796a52 P 0b12969c05f24d038e309045b035f69e: Bootstrap starting.
I20260812 09:48:00.630528 5126 tablet_bootstrap.cc:492] T 2b15723ad5254743b1195fce5a796a52 P a808eee3e1c14f9488cefe0d16ccc66c: Bootstrap starting.
I20260812 09:48:00.632328 5125 tablet_bootstrap.cc:654] T 2b15723ad5254743b1195fce5a796a52 P 0b12969c05f24d038e309045b035f69e: Neither blocks nor log segments found. Creating new log.
I20260812 09:48:00.632810 5126 tablet_bootstrap.cc:654] T 2b15723ad5254743b1195fce5a796a52 P a808eee3e1c14f9488cefe0d16ccc66c: Neither blocks nor log segments found. Creating new log.
I20260812 09:48:00.633308 5125 log.cc:824] T 2b15723ad5254743b1195fce5a796a52 P 0b12969c05f24d038e309045b035f69e: Log is configured to *not* fsync() on all Append() calls
I20260812 09:48:00.633646 5126 log.cc:824] T 2b15723ad5254743b1195fce5a796a52 P a808eee3e1c14f9488cefe0d16ccc66c: Log is configured to *not* fsync() on all Append() calls
I20260812 09:48:00.634537 5127 tablet_bootstrap.cc:492] T 2b15723ad5254743b1195fce5a796a52 P 5cdb8b929df74926b65c64638cbbd3f5: Bootstrap starting.
I20260812 09:48:00.635416 5125 tablet_bootstrap.cc:492] T 2b15723ad5254743b1195fce5a796a52 P 0b12969c05f24d038e309045b035f69e: No bootstrap required, opened a new log
I20260812 09:48:00.635632 5125 ts_tablet_manager.cc:1397] T 2b15723ad5254743b1195fce5a796a52 P 0b12969c05f24d038e309045b035f69e: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260812 09:48:00.635649 5126 tablet_bootstrap.cc:492] T 2b15723ad5254743b1195fce5a796a52 P a808eee3e1c14f9488cefe0d16ccc66c: No bootstrap required, opened a new log
I20260812 09:48:00.635821 5126 ts_tablet_manager.cc:1397] T 2b15723ad5254743b1195fce5a796a52 P a808eee3e1c14f9488cefe0d16ccc66c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260812 09:48:00.636678 5127 tablet_bootstrap.cc:654] T 2b15723ad5254743b1195fce5a796a52 P 5cdb8b929df74926b65c64638cbbd3f5: Neither blocks nor log segments found. Creating new log.
I20260812 09:48:00.637998 5127 log.cc:824] T 2b15723ad5254743b1195fce5a796a52 P 5cdb8b929df74926b65c64638cbbd3f5: Log is configured to *not* fsync() on all Append() calls
I20260812 09:48:00.638691 5125 raft_consensus.cc:348] T 2b15723ad5254743b1195fce5a796a52 P 0b12969c05f24d038e309045b035f69e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cdb8b929df74926b65c64638cbbd3f5" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 32841 } } peers { permanent_uuid: "0b12969c05f24d038e309045b035f69e" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 39701 } } peers { permanent_uuid: "a808eee3e1c14f9488cefe0d16ccc66c" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 43137 } }
I20260812 09:48:00.638983 5125 raft_consensus.cc:374] T 2b15723ad5254743b1195fce5a796a52 P 0b12969c05f24d038e309045b035f69e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:48:00.639042 5125 raft_consensus.cc:729] T 2b15723ad5254743b1195fce5a796a52 P 0b12969c05f24d038e309045b035f69e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b12969c05f24d038e309045b035f69e, State: Initialized, Role: FOLLOWER
I20260812 09:48:00.639496 5125 consensus_queue.cc:260] T 2b15723ad5254743b1195fce5a796a52 P 0b12969c05f24d038e309045b035f69e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cdb8b929df74926b65c64638cbbd3f5" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 32841 } } peers { permanent_uuid: "0b12969c05f24d038e309045b035f69e" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 39701 } } peers { permanent_uuid: "a808eee3e1c14f9488cefe0d16ccc66c" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 43137 } }
I20260812 09:48:00.640012 5127 tablet_bootstrap.cc:492] T 2b15723ad5254743b1195fce5a796a52 P 5cdb8b929df74926b65c64638cbbd3f5: No bootstrap required, opened a new log
I20260812 09:48:00.640187 5127 ts_tablet_manager.cc:1397] T 2b15723ad5254743b1195fce5a796a52 P 5cdb8b929df74926b65c64638cbbd3f5: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260812 09:48:00.640177 5126 raft_consensus.cc:348] T 2b15723ad5254743b1195fce5a796a52 P a808eee3e1c14f9488cefe0d16ccc66c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cdb8b929df74926b65c64638cbbd3f5" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 32841 } } peers { permanent_uuid: "0b12969c05f24d038e309045b035f69e" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 39701 } } peers { permanent_uuid: "a808eee3e1c14f9488cefe0d16ccc66c" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 43137 } }
I20260812 09:48:00.640425 5126 raft_consensus.cc:374] T 2b15723ad5254743b1195fce5a796a52 P a808eee3e1c14f9488cefe0d16ccc66c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:48:00.640504 5126 raft_consensus.cc:729] T 2b15723ad5254743b1195fce5a796a52 P a808eee3e1c14f9488cefe0d16ccc66c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a808eee3e1c14f9488cefe0d16ccc66c, State: Initialized, Role: FOLLOWER
I20260812 09:48:00.640556 5125 ts_tablet_manager.cc:1428] T 2b15723ad5254743b1195fce5a796a52 P 0b12969c05f24d038e309045b035f69e: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260812 09:48:00.640645 4791 heartbeater.cc:503] Master 127.13.99.62:42637 was elected leader, sending a full tablet report...
I20260812 09:48:00.640972 5126 consensus_queue.cc:260] T 2b15723ad5254743b1195fce5a796a52 P a808eee3e1c14f9488cefe0d16ccc66c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cdb8b929df74926b65c64638cbbd3f5" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 32841 } } peers { permanent_uuid: "0b12969c05f24d038e309045b035f69e" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 39701 } } peers { permanent_uuid: "a808eee3e1c14f9488cefe0d16ccc66c" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 43137 } }
I20260812 09:48:00.641633 4537 heartbeater.cc:503] Master 127.13.99.62:42637 was elected leader, sending a full tablet report...
I20260812 09:48:00.641907 5126 ts_tablet_manager.cc:1428] T 2b15723ad5254743b1195fce5a796a52 P a808eee3e1c14f9488cefe0d16ccc66c: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260812 09:48:00.643806 5127 raft_consensus.cc:348] T 2b15723ad5254743b1195fce5a796a52 P 5cdb8b929df74926b65c64638cbbd3f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cdb8b929df74926b65c64638cbbd3f5" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 32841 } } peers { permanent_uuid: "0b12969c05f24d038e309045b035f69e" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 39701 } } peers { permanent_uuid: "a808eee3e1c14f9488cefe0d16ccc66c" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 43137 } }
I20260812 09:48:00.644086 5127 raft_consensus.cc:374] T 2b15723ad5254743b1195fce5a796a52 P 5cdb8b929df74926b65c64638cbbd3f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:48:00.644151 5127 raft_consensus.cc:729] T 2b15723ad5254743b1195fce5a796a52 P 5cdb8b929df74926b65c64638cbbd3f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cdb8b929df74926b65c64638cbbd3f5, State: Initialized, Role: FOLLOWER
I20260812 09:48:00.644588 5127 consensus_queue.cc:260] T 2b15723ad5254743b1195fce5a796a52 P 5cdb8b929df74926b65c64638cbbd3f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cdb8b929df74926b65c64638cbbd3f5" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 32841 } } peers { permanent_uuid: "0b12969c05f24d038e309045b035f69e" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 39701 } } peers { permanent_uuid: "a808eee3e1c14f9488cefe0d16ccc66c" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 43137 } }
I20260812 09:48:00.645247 5088 heartbeater.cc:503] Master 127.13.99.62:42637 was elected leader, sending a full tablet report...
I20260812 09:48:00.645573 5127 ts_tablet_manager.cc:1428] T 2b15723ad5254743b1195fce5a796a52 P 5cdb8b929df74926b65c64638cbbd3f5: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260812 09:48:00.812345 5089 tablet.cc:2369] T 2b15723ad5254743b1195fce5a796a52 P 5cdb8b929df74926b65c64638cbbd3f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 09:48:00.832719 4792 tablet.cc:2369] T 2b15723ad5254743b1195fce5a796a52 P 0b12969c05f24d038e309045b035f69e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 09:48:00.851756 4538 tablet.cc:2369] T 2b15723ad5254743b1195fce5a796a52 P a808eee3e1c14f9488cefe0d16ccc66c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 09:48:00.879968 5137 raft_consensus.cc:482] T 2b15723ad5254743b1195fce5a796a52 P a808eee3e1c14f9488cefe0d16ccc66c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 09:48:00.880234 5137 raft_consensus.cc:504] T 2b15723ad5254743b1195fce5a796a52 P a808eee3e1c14f9488cefe0d16ccc66c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cdb8b929df74926b65c64638cbbd3f5" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 32841 } } peers { permanent_uuid: "0b12969c05f24d038e309045b035f69e" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 39701 } } peers { permanent_uuid: "a808eee3e1c14f9488cefe0d16ccc66c" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 43137 } }
I20260812 09:48:00.881477 5137 leader_election.cc:288] T 2b15723ad5254743b1195fce5a796a52 P a808eee3e1c14f9488cefe0d16ccc66c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5cdb8b929df74926b65c64638cbbd3f5 (127.13.99.3:32841), 0b12969c05f24d038e309045b035f69e (127.13.99.2:39701)
I20260812 09:48:00.885617 5120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:00.881903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.1:48321 (local address 127.13.99.2:39701)
0812 09:48:00.882069 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0812 09:48:00.882074 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:48:00.882339 (+ 265us) server_negotiation.cc:407] Connection header received
0812 09:48:00.882534 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:48:00.882542 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:48:00.882584 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:48:00.882665 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:48:00.882957 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:48:00.883475 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:48:00.884635 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:48:00.885034 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:48:00.885155 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:48:00.885221 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0812 09:48:00.885420 (+ 199us) server_negotiation.cc:299] Negotiation successful
0812 09:48:00.885507 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260812 09:48:00.885812 5119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:00.881916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.1:32777 (local address 127.13.99.3:32841)
0812 09:48:00.882073 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0812 09:48:00.882077 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:48:00.882506 (+ 429us) server_negotiation.cc:407] Connection header received
0812 09:48:00.882668 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:48:00.882671 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:48:00.882724 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:48:00.882813 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:48:00.883193 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:48:00.883711 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:48:00.884801 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:48:00.885231 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:48:00.885355 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:48:00.885421 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0812 09:48:00.885621 (+ 200us) server_negotiation.cc:299] Negotiation successful
0812 09:48:00.885692 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260812 09:48:00.885644 5141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:00.881784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.2:39701 (local address 127.13.99.1:48321)
0812 09:48:00.882232 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0812 09:48:00.882244 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 09:48:00.882407 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:48:00.882692 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:48:00.882695 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:48:00.882711 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:48:00.882840 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:48:00.882846 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:48:00.883588 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:48:00.883594 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:48:00.884452 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:48:00.884462 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:48:00.885165 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:48:00.885168 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:48:00.885268 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:48:00.885292 (+ 24us) client_negotiation.cc:769] Sending connection context
0812 09:48:00.885389 (+ 97us) client_negotiation.cc:240] Negotiation successful
0812 09:48:00.885469 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"mutex_wait_us":22,"thread_start_us":162,"threads_started":1}
I20260812 09:48:00.885830 5142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:00.881784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.3:32841 (local address 127.13.99.1:32777)
0812 09:48:00.882385 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0812 09:48:00.882398 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 09:48:00.882538 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:48:00.882827 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:48:00.882830 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:48:00.882847 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:48:00.882985 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:48:00.882991 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:48:00.883848 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:48:00.883853 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:48:00.884668 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:48:00.884678 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:48:00.885362 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:48:00.885366 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:48:00.885488 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:48:00.885508 (+ 20us) client_negotiation.cc:769] Sending connection context
0812 09:48:00.885602 (+ 94us) client_negotiation.cc:240] Negotiation successful
0812 09:48:00.885673 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"mutex_wait_us":184,"thread_start_us":103,"threads_started":1}
I20260812 09:48:00.886665 4694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b15723ad5254743b1195fce5a796a52" candidate_uuid: "a808eee3e1c14f9488cefe0d16ccc66c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b12969c05f24d038e309045b035f69e" is_pre_election: true
I20260812 09:48:00.886662 4973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b15723ad5254743b1195fce5a796a52" candidate_uuid: "a808eee3e1c14f9488cefe0d16ccc66c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cdb8b929df74926b65c64638cbbd3f5" is_pre_election: true
I20260812 09:48:00.887002 4973 raft_consensus.cc:2454] T 2b15723ad5254743b1195fce5a796a52 P 5cdb8b929df74926b65c64638cbbd3f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a808eee3e1c14f9488cefe0d16ccc66c in term 0.
I20260812 09:48:00.887033 4694 raft_consensus.cc:2454] T 2b15723ad5254743b1195fce5a796a52 P 0b12969c05f24d038e309045b035f69e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a808eee3e1c14f9488cefe0d16ccc66c in term 0.
I20260812 09:48:00.887583 4331 leader_election.cc:302] T 2b15723ad5254743b1195fce5a796a52 P a808eee3e1c14f9488cefe0d16ccc66c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0b12969c05f24d038e309045b035f69e, 5cdb8b929df74926b65c64638cbbd3f5, a808eee3e1c14f9488cefe0d16ccc66c; no voters:
I20260812 09:48:00.887903 5137 raft_consensus.cc:2781] T 2b15723ad5254743b1195fce5a796a52 P a808eee3e1c14f9488cefe0d16ccc66c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 09:48:00.888000 5137 raft_consensus.cc:482] T 2b15723ad5254743b1195fce5a796a52 P a808eee3e1c14f9488cefe0d16ccc66c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 09:48:00.888048 5137 raft_consensus.cc:3037] T 2b15723ad5254743b1195fce5a796a52 P a808eee3e1c14f9488cefe0d16ccc66c [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:48:00.889219 5137 raft_consensus.cc:504] T 2b15723ad5254743b1195fce5a796a52 P a808eee3e1c14f9488cefe0d16ccc66c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cdb8b929df74926b65c64638cbbd3f5" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 32841 } } peers { permanent_uuid: "0b12969c05f24d038e309045b035f69e" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 39701 } } peers { permanent_uuid: "a808eee3e1c14f9488cefe0d16ccc66c" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 43137 } }
I20260812 09:48:00.889592 5137 leader_election.cc:288] T 2b15723ad5254743b1195fce5a796a52 P a808eee3e1c14f9488cefe0d16ccc66c [CANDIDATE]: Term 1 election: Requested vote from peers 5cdb8b929df74926b65c64638cbbd3f5 (127.13.99.3:32841), 0b12969c05f24d038e309045b035f69e (127.13.99.2:39701)
I20260812 09:48:00.889995 4694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b15723ad5254743b1195fce5a796a52" candidate_uuid: "a808eee3e1c14f9488cefe0d16ccc66c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b12969c05f24d038e309045b035f69e"
I20260812 09:48:00.890012 4973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b15723ad5254743b1195fce5a796a52" candidate_uuid: "a808eee3e1c14f9488cefe0d16ccc66c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cdb8b929df74926b65c64638cbbd3f5"
I20260812 09:48:00.890137 4694 raft_consensus.cc:3037] T 2b15723ad5254743b1195fce5a796a52 P 0b12969c05f24d038e309045b035f69e [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:48:00.890152 4973 raft_consensus.cc:3037] T 2b15723ad5254743b1195fce5a796a52 P 5cdb8b929df74926b65c64638cbbd3f5 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:48:00.891240 4694 raft_consensus.cc:2454] T 2b15723ad5254743b1195fce5a796a52 P 0b12969c05f24d038e309045b035f69e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a808eee3e1c14f9488cefe0d16ccc66c in term 1.
I20260812 09:48:00.891304 4973 raft_consensus.cc:2454] T 2b15723ad5254743b1195fce5a796a52 P 5cdb8b929df74926b65c64638cbbd3f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a808eee3e1c14f9488cefe0d16ccc66c in term 1.
I20260812 09:48:00.891567 4331 leader_election.cc:302] T 2b15723ad5254743b1195fce5a796a52 P a808eee3e1c14f9488cefe0d16ccc66c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b12969c05f24d038e309045b035f69e, a808eee3e1c14f9488cefe0d16ccc66c; no voters:
I20260812 09:48:00.891754 5137 raft_consensus.cc:2781] T 2b15723ad5254743b1195fce5a796a52 P a808eee3e1c14f9488cefe0d16ccc66c [term 1 FOLLOWER]: Leader election won for term 1
I20260812 09:48:00.892023 5137 raft_consensus.cc:686] T 2b15723ad5254743b1195fce5a796a52 P a808eee3e1c14f9488cefe0d16ccc66c [term 1 LEADER]: Becoming Leader. State: Replica: a808eee3e1c14f9488cefe0d16ccc66c, State: Running, Role: LEADER
I20260812 09:48:00.892374 5137 consensus_queue.cc:237] T 2b15723ad5254743b1195fce5a796a52 P a808eee3e1c14f9488cefe0d16ccc66c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cdb8b929df74926b65c64638cbbd3f5" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 32841 } } peers { permanent_uuid: "0b12969c05f24d038e309045b035f69e" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 39701 } } peers { permanent_uuid: "a808eee3e1c14f9488cefe0d16ccc66c" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 43137 } }
I20260812 09:48:00.895700 4166 catalog_manager.cc:5495] T 2b15723ad5254743b1195fce5a796a52 P a808eee3e1c14f9488cefe0d16ccc66c reported cstate change: term changed from 0 to 1, leader changed from <none> to a808eee3e1c14f9488cefe0d16ccc66c (127.13.99.1). New cstate: current_term: 1 leader_uuid: "a808eee3e1c14f9488cefe0d16ccc66c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cdb8b929df74926b65c64638cbbd3f5" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 32841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b12969c05f24d038e309045b035f69e" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 39701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a808eee3e1c14f9488cefe0d16ccc66c" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 43137 } health_report { overall_health: HEALTHY } } }
I20260812 09:48:00.912647 5123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:00.909010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45892 (local address 127.13.99.1:43137)
0812 09:48:00.909158 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0812 09:48:00.909162 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:48:00.909176 (+ 14us) server_negotiation.cc:407] Connection header received
0812 09:48:00.909350 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:48:00.909354 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:48:00.909409 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:48:00.909524 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 09:48:00.909773 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:48:00.910287 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:48:00.911159 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:48:00.911385 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:48:00.911498 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:48:00.911808 (+ 310us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 09:48:00.912213 (+ 405us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 09:48:00.912349 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0812 09:48:00.912418 (+ 69us) server_negotiation.cc:299] Negotiation successful
0812 09:48:00.912498 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 09:48:00.919026 4973 raft_consensus.cc:1261] T 2b15723ad5254743b1195fce5a796a52 P 5cdb8b929df74926b65c64638cbbd3f5 [term 1 FOLLOWER]: Refusing update from remote peer a808eee3e1c14f9488cefe0d16ccc66c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 09:48:00.919085 4694 raft_consensus.cc:1261] T 2b15723ad5254743b1195fce5a796a52 P 0b12969c05f24d038e309045b035f69e [term 1 FOLLOWER]: Refusing update from remote peer a808eee3e1c14f9488cefe0d16ccc66c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 09:48:00.919739 5145 consensus_queue.cc:1035] T 2b15723ad5254743b1195fce5a796a52 P a808eee3e1c14f9488cefe0d16ccc66c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cdb8b929df74926b65c64638cbbd3f5" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 32841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 09:48:00.919948 5137 consensus_queue.cc:1035] T 2b15723ad5254743b1195fce5a796a52 P a808eee3e1c14f9488cefe0d16ccc66c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b12969c05f24d038e309045b035f69e" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 39701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 09:48:00.930033 5155 mvcc.cc:205] Tried to move back new op lower bound from 7317619019435888640 to 7317619019338403840. Current Snapshot: MvccSnapshot[applied={T|T < 7317619019435888640}]
I20260812 09:48:00.932785 5151 mvcc.cc:205] Tried to move back new op lower bound from 7317619019435888640 to 7317619019338403840. Current Snapshot: MvccSnapshot[applied={T|T < 7317619019435888640 or (T in {7317619019435888640})}]
I20260812 09:48:00.935556 4165 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49476:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 09:48:00.935739 4165 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49476:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 09:48:00.938298 4165 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 83d811857241469db629144445c55d66: Sending DeleteTablet for 3 replicas of tablet 2b15723ad5254743b1195fce5a796a52
I20260812 09:48:00.939229 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 4297
I20260812 09:48:00.939193 4648 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b15723ad5254743b1195fce5a796a52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 09:48:00 UTC) from {username='kudu', principal='kudu/127.13.99.62@KRBTEST.COM'} at 127.0.0.1:59154
I20260812 09:48:00.939334 4929 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b15723ad5254743b1195fce5a796a52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 09:48:00 UTC) from {username='kudu', principal='kudu/127.13.99.62@KRBTEST.COM'} at 127.0.0.1:34668
I20260812 09:48:00.948019 5166 tablet_replica.cc:330] T 2b15723ad5254743b1195fce5a796a52 P 0b12969c05f24d038e309045b035f69e: stopping tablet replica
I20260812 09:48:00.948417 5166 raft_consensus.cc:2229] T 2b15723ad5254743b1195fce5a796a52 P 0b12969c05f24d038e309045b035f69e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 09:48:00.948805 5166 raft_consensus.cc:2258] T 2b15723ad5254743b1195fce5a796a52 P 0b12969c05f24d038e309045b035f69e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 09:48:00.950284 5166 ts_tablet_manager.cc:1905] T 2b15723ad5254743b1195fce5a796a52 P 0b12969c05f24d038e309045b035f69e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 09:48:00.952611 5166 ts_tablet_manager.cc:1918] T 2b15723ad5254743b1195fce5a796a52 P 0b12969c05f24d038e309045b035f69e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 09:48:00.952698 5166 log.cc:1196] T 2b15723ad5254743b1195fce5a796a52 P 0b12969c05f24d038e309045b035f69e: Deleting WAL directory at /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-1/wal/wals/2b15723ad5254743b1195fce5a796a52
I20260812 09:48:00.952947 5167 tablet_replica.cc:330] T 2b15723ad5254743b1195fce5a796a52 P 5cdb8b929df74926b65c64638cbbd3f5: stopping tablet replica
I20260812 09:48:00.953056 5166 ts_tablet_manager.cc:1939] T 2b15723ad5254743b1195fce5a796a52 P 0b12969c05f24d038e309045b035f69e: Deleting consensus metadata
I20260812 09:48:00.953294 5167 raft_consensus.cc:2229] T 2b15723ad5254743b1195fce5a796a52 P 5cdb8b929df74926b65c64638cbbd3f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 09:48:00.953629 5167 raft_consensus.cc:2258] T 2b15723ad5254743b1195fce5a796a52 P 5cdb8b929df74926b65c64638cbbd3f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 09:48:00.954371 4144 catalog_manager.cc:4841] TS 0b12969c05f24d038e309045b035f69e (127.13.99.2:39701): tablet 2b15723ad5254743b1195fce5a796a52 (table test-table [id=f562716082a34e278473bf195c2fa19f]) successfully deleted
I20260812 09:48:00.955286 5167 ts_tablet_manager.cc:1905] T 2b15723ad5254743b1195fce5a796a52 P 5cdb8b929df74926b65c64638cbbd3f5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260812 09:48:00.955956 4144 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.99.1:43137 (error 108)
I20260812 09:48:00.956240 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 4544
I20260812 09:48:00.961133 5167 ts_tablet_manager.cc:1918] T 2b15723ad5254743b1195fce5a796a52 P 5cdb8b929df74926b65c64638cbbd3f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 09:48:00.961236 5167 log.cc:1196] T 2b15723ad5254743b1195fce5a796a52 P 5cdb8b929df74926b65c64638cbbd3f5: Deleting WAL directory at /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-2/wal/wals/2b15723ad5254743b1195fce5a796a52
I20260812 09:48:00.961539 5167 ts_tablet_manager.cc:1939] T 2b15723ad5254743b1195fce5a796a52 P 5cdb8b929df74926b65c64638cbbd3f5: Deleting consensus metadata
I20260812 09:48:00.962738 5109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:00.962390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.1:43137 (local address 127.0.0.1:45906)
0812 09:48:00.962544 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0812 09:48:00.962632 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.1:43137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260812 09:48:00.962934 4144 catalog_manager.cc:4584] TS a808eee3e1c14f9488cefe0d16ccc66c (127.13.99.1:43137): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2b15723ad5254743b1195fce5a796a52: Network error: Client connection negotiation failed: client connection to 127.13.99.1:43137: connect: Connection refused (error 111)
I20260812 09:48:00.963321 4145 catalog_manager.cc:4841] TS 5cdb8b929df74926b65c64638cbbd3f5 (127.13.99.3:32841): tablet 2b15723ad5254743b1195fce5a796a52 (table test-table [id=f562716082a34e278473bf195c2fa19f]) successfully deleted
I20260812 09:48:00.968370 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 4797
I20260812 09:48:00.977876 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 4116
I20260812 09:48:00.992651 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.62:42637
--webserver_interface=127.13.99.62
--webserver_port=40349
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab
--principal=kudu/127.13.99.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:33335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.99.62:42637
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:48:01.105435 5170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:48:01.105701 5170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:48:01.105779 5170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:48:01.109745 5170 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 09:48:01.109824 5170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:48:01.109858 5170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:48:01.109898 5170 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 09:48:01.109934 5170 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 09:48:01.115051 5170 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:33335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.13.99.62:42637
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab
--principal=kudu/127.13.99.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.62:42637
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.99.62
--webserver_port=40349
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5170
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:48:01.116084 5170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:48:01.117130 5170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:48:01.131018 5183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:48:01.131018 5187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:48:01.131045 5185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:48:01.131381 5170 server_base.cc:1034] running on GCE node
I20260812 09:48:01.132005 5170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:48:01.133231 5170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:48:01.134404 5170 hybrid_clock.cc:648] HybridClock initialized: now 1786528081134387 us; error 27 us; skew 500 ppm
Aug 12 09:48:01 dist-test-slave-jq4t krb5kdc[4086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528081, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:48:01.137320 5170 init.cc:377] Logged in from keytab as kudu/127.13.99.62@KRBTEST.COM (short username kudu)
I20260812 09:48:01.138245 5170 webserver.cc:460] Webserver started at http://127.13.99.62:40349/ using document root <none> and password file <none>
I20260812 09:48:01.138840 5170 fs_manager.cc:359] Metadata directory not provided
I20260812 09:48:01.138942 5170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:48:01.142886 5170 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260812 09:48:01.145088 5199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:48:01.146297 5170 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 09:48:01.146461 5170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/master-0/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/master-0/wal
uuid: "83d811857241469db629144445c55d66"
format_stamp: "Formatted at 2026-08-12 09:47:59 on dist-test-slave-jq4t"
server_key: "fdf1de0a3059815d8e6a12d9b4a6761a"
server_key_iv: "78086bf54cfa847ca62731e158cbefed"
server_key_version: "encryptionkey@0"
I20260812 09:48:01.147048 5170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:48:01.158440 5170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:48:01.161674 5170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:48:01.161878 5170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:48:01.169531 5170 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.62:42637
I20260812 09:48:01.169914 5318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.62:42637 every 8 connection(s)
I20260812 09:48:01.170682 5170 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
I20260812 09:48:01.175128 5319 sys_catalog.cc:263] Verifying existing consensus state
I20260812 09:48:01.176849 5319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83d811857241469db629144445c55d66: Bootstrap starting.
I20260812 09:48:01.179522 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 5170
I20260812 09:48:01.180073 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.99.1:43137
--local_ip_for_outbound_sockets=127.13.99.1
--tserver_master_addrs=127.13.99.62:42637
--webserver_port=37531
--webserver_interface=127.13.99.1
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.1.keytab
--principal=kudu/127.13.99.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:33335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 09:48:01.196182 5319 log.cc:824] T 00000000000000000000000000000000 P 83d811857241469db629144445c55d66: Log is configured to *not* fsync() on all Append() calls
I20260812 09:48:01.206125 5319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83d811857241469db629144445c55d66: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 09:48:01.206540 5319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83d811857241469db629144445c55d66: Bootstrap complete.
I20260812 09:48:01.211177 5319 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83d811857241469db629144445c55d66 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83d811857241469db629144445c55d66" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 42637 } }
I20260812 09:48:01.211587 5319 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83d811857241469db629144445c55d66 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83d811857241469db629144445c55d66, State: Initialized, Role: FOLLOWER
I20260812 09:48:01.212133 5319 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83d811857241469db629144445c55d66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83d811857241469db629144445c55d66" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 42637 } }
I20260812 09:48:01.212288 5319 raft_consensus.cc:388] T 00000000000000000000000000000000 P 83d811857241469db629144445c55d66 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 09:48:01.212366 5319 raft_consensus.cc:482] T 00000000000000000000000000000000 P 83d811857241469db629144445c55d66 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 09:48:01.212517 5319 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83d811857241469db629144445c55d66 [term 1 FOLLOWER]: Advancing to term 2
I20260812 09:48:01.214473 5319 raft_consensus.cc:504] T 00000000000000000000000000000000 P 83d811857241469db629144445c55d66 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83d811857241469db629144445c55d66" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 42637 } }
I20260812 09:48:01.214843 5319 leader_election.cc:302] T 00000000000000000000000000000000 P 83d811857241469db629144445c55d66 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 83d811857241469db629144445c55d66; no voters:
I20260812 09:48:01.215194 5319 leader_election.cc:288] T 00000000000000000000000000000000 P 83d811857241469db629144445c55d66 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260812 09:48:01.215281 5325 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 83d811857241469db629144445c55d66 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 09:48:01.215479 5325 raft_consensus.cc:686] T 00000000000000000000000000000000 P 83d811857241469db629144445c55d66 [term 2 LEADER]: Becoming Leader. State: Replica: 83d811857241469db629144445c55d66, State: Running, Role: LEADER
I20260812 09:48:01.215876 5325 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83d811857241469db629144445c55d66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83d811857241469db629144445c55d66" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 42637 } }
I20260812 09:48:01.216564 5319 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83d811857241469db629144445c55d66 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 09:48:01.217316 5327 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83d811857241469db629144445c55d66 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "83d811857241469db629144445c55d66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83d811857241469db629144445c55d66" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 42637 } } }
I20260812 09:48:01.217461 5327 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83d811857241469db629144445c55d66 [sys.catalog]: This master's current role is: LEADER
I20260812 09:48:01.217774 5328 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83d811857241469db629144445c55d66 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83d811857241469db629144445c55d66. Latest consensus state: current_term: 2 leader_uuid: "83d811857241469db629144445c55d66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83d811857241469db629144445c55d66" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 42637 } } }
I20260812 09:48:01.217926 5328 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83d811857241469db629144445c55d66 [sys.catalog]: This master's current role is: LEADER
I20260812 09:48:01.218288 5341 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 09:48:01.223335 5341 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 09:48:01.224035 5341 catalog_manager.cc:1259] Loaded cluster ID: b991061c7a9e413fa211af0ca6b9e99c
I20260812 09:48:01.224098 5341 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 09:48:01.225592 5341 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260812 09:48:01.230082 5341 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260812 09:48:01.235728 5341 catalog_manager.cc:1504] Loading token signing keys...
I20260812 09:48:01.236528 5341 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 83d811857241469db629144445c55d66: Loaded TSK: 0
I20260812 09:48:01.237733 5341 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 09:48:01.305891 5322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:48:01.306150 5322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:48:01.306195 5322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:48:01.310285 5322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:48:01.310353 5322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:48:01.310411 5322 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.99.1
I20260812 09:48:01.315788 5322 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:33335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.1.keytab
--principal=kudu/127.13.99.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.99.1:43137
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.99.1
--webserver_port=37531
--tserver_master_addrs=127.13.99.62:42637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5322
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.99.1
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:48:01.316959 5322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:48:01.318121 5322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:48:01.328306 5364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:48:01.328302 5371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:48:01.328923 5375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:48:01.329063 5322 server_base.cc:1034] running on GCE node
I20260812 09:48:01.329522 5322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:48:01.330459 5322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:48:01.331696 5322 hybrid_clock.cc:648] HybridClock initialized: now 1786528081331670 us; error 39 us; skew 500 ppm
Aug 12 09:48:01 dist-test-slave-jq4t krb5kdc[4086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528081, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:48:01.335418 5322 init.cc:377] Logged in from keytab as kudu/127.13.99.1@KRBTEST.COM (short username kudu)
I20260812 09:48:01.336659 5322 webserver.cc:460] Webserver started at http://127.13.99.1:37531/ using document root <none> and password file <none>
I20260812 09:48:01.337448 5322 fs_manager.cc:359] Metadata directory not provided
I20260812 09:48:01.337575 5322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:48:01.342965 5322 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260812 09:48:01.345762 5388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:48:01.347311 5322 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 09:48:01.347451 5322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-0/wal
uuid: "a808eee3e1c14f9488cefe0d16ccc66c"
format_stamp: "Formatted at 2026-08-12 09:48:00 on dist-test-slave-jq4t"
server_key: "cc68ccaa2e555cece62a47eb75f4b47c"
server_key_iv: "5419744aa6331d0fba4b0e14ae3946db"
server_key_version: "encryptionkey@0"
I20260812 09:48:01.348124 5322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:48:01.374752 5322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:48:01.378571 5322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:48:01.378732 5322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:48:01.379356 5322 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 09:48:01.380715 5403 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260812 09:48:01.386991 5322 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 09:48:01.387058 5322 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.000s sys 0.000s
I20260812 09:48:01.387140 5322 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260812 09:48:01.389330 5322 ts_tablet_manager.cc:610] Registered 1 tablets
I20260812 09:48:01.389401 5322 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260812 09:48:01.389791 5403 tablet_bootstrap.cc:492] T 2b15723ad5254743b1195fce5a796a52 P a808eee3e1c14f9488cefe0d16ccc66c: Bootstrap starting.
I20260812 09:48:01.400616 5322 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.1:43137
I20260812 09:48:01.401845 5322 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-0/data/info.pb
I20260812 09:48:01.402163 5621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.1:43137 every 8 connection(s)
I20260812 09:48:01.408820 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 5322
I20260812 09:48:01.409305 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.99.2:39701
--local_ip_for_outbound_sockets=127.13.99.2
--tserver_master_addrs=127.13.99.62:42637
--webserver_port=35313
--webserver_interface=127.13.99.2
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.2.keytab
--principal=kudu/127.13.99.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:33335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 09:48:01.411463 5403 log.cc:824] T 2b15723ad5254743b1195fce5a796a52 P a808eee3e1c14f9488cefe0d16ccc66c: Log is configured to *not* fsync() on all Append() calls
Aug 12 09:48:01 dist-test-slave-jq4t krb5kdc[4086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528081, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.1@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
I20260812 09:48:01.420312 5403 tablet_bootstrap.cc:492] T 2b15723ad5254743b1195fce5a796a52 P a808eee3e1c14f9488cefe0d16ccc66c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 09:48:01.420749 5403 tablet_bootstrap.cc:492] T 2b15723ad5254743b1195fce5a796a52 P a808eee3e1c14f9488cefe0d16ccc66c: Bootstrap complete.
I20260812 09:48:01.421463 5624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:01.403899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:42637 (local address 127.13.99.1:44055)
0812 09:48:01.404428 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0812 09:48:01.404512 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0812 09:48:01.405351 (+ 839us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:48:01.406843 (+ 1492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:48:01.406850 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:48:01.407318 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:48:01.407789 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:48:01.407805 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:48:01.409823 (+ 2018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:48:01.409829 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:48:01.411330 (+ 1501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:48:01.411346 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:48:01.412279 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:48:01.412285 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:48:01.412562 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:48:01.413293 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:48:01.413320 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:48:01.414958 (+ 1638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:48:01.417870 (+ 2912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:48:01.417876 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:48:01.417889 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:48:01.418222 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:48:01.418661 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:48:01.418664 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:48:01.418666 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:48:01.418849 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:48:01.419637 (+ 788us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:48:01.419643 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:48:01.419960 (+ 317us) client_negotiation.cc:769] Sending connection context
0812 09:48:01.420161 (+ 201us) client_negotiation.cc:240] Negotiation successful
0812 09:48:01.420528 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":166,"threads_started":1}
I20260812 09:48:01.421646 5626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:01.404022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.1:44055 (local address 127.13.99.62:42637)
0812 09:48:01.404377 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0812 09:48:01.404396 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0812 09:48:01.404639 (+ 243us) server_negotiation.cc:407] Connection header received
0812 09:48:01.405829 (+ 1190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:48:01.405846 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:48:01.406336 (+ 490us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:48:01.406767 (+ 431us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:48:01.407993 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:48:01.409644 (+ 1651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:48:01.411481 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:48:01.412108 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:48:01.412287 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:48:01.415162 (+ 2875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:48:01.415171 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:48:01.415190 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:48:01.415227 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:48:01.417715 (+ 2488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:48:01.418357 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:48:01.418366 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:48:01.418373 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:48:01.418542 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:48:01.418965 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:48:01.418971 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:48:01.418974 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:48:01.419495 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:48:01.419654 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0812 09:48:01.420329 (+ 675us) server_negotiation.cc:299] Negotiation successful
0812 09:48:01.420703 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":129,"threads_started":1}
I20260812 09:48:01.422149 5403 ts_tablet_manager.cc:1397] T 2b15723ad5254743b1195fce5a796a52 P a808eee3e1c14f9488cefe0d16ccc66c: Time spent bootstrapping tablet: real 0.033s user 0.022s sys 0.007s
I20260812 09:48:01.424103 5622 heartbeater.cc:347] Connected to a master server at 127.13.99.62:42637
I20260812 09:48:01.424319 5622 heartbeater.cc:464] Registering TS with master...
I20260812 09:48:01.424934 5622 heartbeater.cc:511] Master 127.13.99.62:42637 requested a full tablet report, sending...
I20260812 09:48:01.425015 5403 raft_consensus.cc:348] T 2b15723ad5254743b1195fce5a796a52 P a808eee3e1c14f9488cefe0d16ccc66c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cdb8b929df74926b65c64638cbbd3f5" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 32841 } } peers { permanent_uuid: "0b12969c05f24d038e309045b035f69e" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 39701 } } peers { permanent_uuid: "a808eee3e1c14f9488cefe0d16ccc66c" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 43137 } }
I20260812 09:48:01.425520 5403 raft_consensus.cc:729] T 2b15723ad5254743b1195fce5a796a52 P a808eee3e1c14f9488cefe0d16ccc66c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a808eee3e1c14f9488cefe0d16ccc66c, State: Initialized, Role: FOLLOWER
I20260812 09:48:01.425981 5403 consensus_queue.cc:260] T 2b15723ad5254743b1195fce5a796a52 P a808eee3e1c14f9488cefe0d16ccc66c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cdb8b929df74926b65c64638cbbd3f5" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 32841 } } peers { permanent_uuid: "0b12969c05f24d038e309045b035f69e" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 39701 } } peers { permanent_uuid: "a808eee3e1c14f9488cefe0d16ccc66c" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 43137 } }
I20260812 09:48:01.427049 5403 ts_tablet_manager.cc:1428] T 2b15723ad5254743b1195fce5a796a52 P a808eee3e1c14f9488cefe0d16ccc66c: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260812 09:48:01.428943 5242 ts_manager.cc:194] Registered new tserver with Master: a808eee3e1c14f9488cefe0d16ccc66c (127.13.99.1:43137)
I20260812 09:48:01.432190 5242 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.99.1@KRBTEST.COM'} at 127.13.99.1:44055
I20260812 09:48:01.434644 5622 heartbeater.cc:503] Master 127.13.99.62:42637 was elected leader, sending a full tablet report...
Aug 12 09:48:01 dist-test-slave-jq4t krb5kdc[4086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528081, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.62@KRBTEST.COM for kudu/127.13.99.1@KRBTEST.COM
I20260812 09:48:01.442811 5632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:01.431717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45920 (local address 127.13.99.1:43137)
0812 09:48:01.432095 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0812 09:48:01.432106 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 09:48:01.432514 (+ 408us) server_negotiation.cc:407] Connection header received
0812 09:48:01.432731 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:48:01.432738 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:48:01.432944 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:48:01.433072 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:48:01.433826 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:48:01.435055 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:48:01.436290 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:48:01.436680 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:48:01.436766 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:48:01.439271 (+ 2505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:48:01.439280 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:48:01.439290 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:48:01.439312 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:48:01.440991 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:48:01.441517 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:48:01.441522 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:48:01.441523 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:48:01.441598 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:48:01.441875 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:48:01.441877 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:48:01.441878 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:48:01.442095 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:48:01.442194 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0812 09:48:01.442561 (+ 367us) server_negotiation.cc:299] Negotiation successful
0812 09:48:01.442692 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":107,"threads_started":1}
I20260812 09:48:01.442991 5630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:01.431610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.1:43137 (local address 127.0.0.1:45920)
0812 09:48:01.432354 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0812 09:48:01.432396 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 09:48:01.432575 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:48:01.433137 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:48:01.433144 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:48:01.433570 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:48:01.433695 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:48:01.433704 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:48:01.435200 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:48:01.435204 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:48:01.436175 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:48:01.436187 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:48:01.436817 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:48:01.436821 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:48:01.436933 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:48:01.437465 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:48:01.437481 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:48:01.439132 (+ 1651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:48:01.441100 (+ 1968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:48:01.441110 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:48:01.441115 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:48:01.441404 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:48:01.441687 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:48:01.441690 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:48:01.441692 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:48:01.441788 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:48:01.442208 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:48:01.442214 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:48:01.442338 (+ 124us) client_negotiation.cc:769] Sending connection context
0812 09:48:01.442490 (+ 152us) client_negotiation.cc:240] Negotiation successful
0812 09:48:01.442685 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":102,"threads_started":1}
I20260812 09:48:01.443998 5468 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b15723ad5254743b1195fce5a796a52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 09:48:00 UTC) from {username='kudu', principal='kudu/127.13.99.62@KRBTEST.COM'} at 127.0.0.1:45920
I20260812 09:48:01.444963 5633 tablet_replica.cc:330] stopping tablet replica
I20260812 09:48:01.445255 5633 raft_consensus.cc:2229] T 2b15723ad5254743b1195fce5a796a52 P a808eee3e1c14f9488cefe0d16ccc66c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 09:48:01.445513 5633 raft_consensus.cc:2258] T 2b15723ad5254743b1195fce5a796a52 P a808eee3e1c14f9488cefe0d16ccc66c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 09:48:01.446480 5633 ts_tablet_manager.cc:1905] T 2b15723ad5254743b1195fce5a796a52 P a808eee3e1c14f9488cefe0d16ccc66c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 09:48:01.450088 5633 ts_tablet_manager.cc:1918] T 2b15723ad5254743b1195fce5a796a52 P a808eee3e1c14f9488cefe0d16ccc66c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 09:48:01.450218 5633 log.cc:1196] T 2b15723ad5254743b1195fce5a796a52 P a808eee3e1c14f9488cefe0d16ccc66c: Deleting WAL directory at /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-0/wal/wals/2b15723ad5254743b1195fce5a796a52
I20260812 09:48:01.450524 5633 ts_tablet_manager.cc:1939] T 2b15723ad5254743b1195fce5a796a52 P a808eee3e1c14f9488cefe0d16ccc66c: Deleting consensus metadata
I20260812 09:48:01.452405 5208 catalog_manager.cc:4841] TS a808eee3e1c14f9488cefe0d16ccc66c (127.13.99.1:43137): tablet 2b15723ad5254743b1195fce5a796a52 (table test-table [id=f562716082a34e278473bf195c2fa19f]) successfully deleted
W20260812 09:48:01.531308 5627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:48:01.531548 5627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:48:01.531589 5627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:48:01.535547 5627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:48:01.535598 5627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:48:01.535653 5627 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.99.2
I20260812 09:48:01.540656 5627 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:33335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.2.keytab
--principal=kudu/127.13.99.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.99.2:39701
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.99.2
--webserver_port=35313
--tserver_master_addrs=127.13.99.62:42637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5627
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.99.2
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:48:01.541544 5627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:48:01.542450 5627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:48:01.551903 5651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:48:01.551882 5643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:48:01.551929 5648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:48:01.552101 5627 server_base.cc:1034] running on GCE node
I20260812 09:48:01.552775 5627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:48:01.553440 5627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:48:01.554601 5627 hybrid_clock.cc:648] HybridClock initialized: now 1786528081554573 us; error 49 us; skew 500 ppm
Aug 12 09:48:01 dist-test-slave-jq4t krb5kdc[4086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528081, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:48:01.557394 5627 init.cc:377] Logged in from keytab as kudu/127.13.99.2@KRBTEST.COM (short username kudu)
I20260812 09:48:01.558259 5627 webserver.cc:460] Webserver started at http://127.13.99.2:35313/ using document root <none> and password file <none>
I20260812 09:48:01.558831 5627 fs_manager.cc:359] Metadata directory not provided
I20260812 09:48:01.558941 5627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:48:01.563062 5627 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20260812 09:48:01.565121 5663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:48:01.566288 5627 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20260812 09:48:01.566443 5627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-1/wal
uuid: "0b12969c05f24d038e309045b035f69e"
format_stamp: "Formatted at 2026-08-12 09:48:00 on dist-test-slave-jq4t"
server_key: "4ae9a995fc90c118a5b17e256f83c06f"
server_key_iv: "6c459649e7869d8cb1627147f978999d"
server_key_version: "encryptionkey@0"
I20260812 09:48:01.566965 5627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:48:01.579833 5627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:48:01.582749 5627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:48:01.582948 5627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:48:01.583527 5627 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 09:48:01.584337 5627 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:48:01.584411 5627 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 09:48:01.584477 5627 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:48:01.584553 5627 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 09:48:01.594231 5627 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.2:39701
I20260812 09:48:01.594254 5880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.2:39701 every 8 connection(s)
I20260812 09:48:01.595268 5627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-1/data/info.pb
I20260812 09:48:01.599351 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 5627
I20260812 09:48:01.599846 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.99.3:32841
--local_ip_for_outbound_sockets=127.13.99.3
--tserver_master_addrs=127.13.99.62:42637
--webserver_port=43579
--webserver_interface=127.13.99.3
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.3.keytab
--principal=kudu/127.13.99.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:33335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 09:48:01 dist-test-slave-jq4t krb5kdc[4086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528081, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.2@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
I20260812 09:48:01.611878 5626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:01.597211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.2:52753 (local address 127.13.99.62:42637)
0812 09:48:01.597387 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0812 09:48:01.597393 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 09:48:01.598296 (+ 903us) server_negotiation.cc:407] Connection header received
0812 09:48:01.599740 (+ 1444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:48:01.599747 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:48:01.599817 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:48:01.599925 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:48:01.601228 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:48:01.602021 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:48:01.603831 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:48:01.604062 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:48:01.604146 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:48:01.607230 (+ 3084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:48:01.607236 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:48:01.607243 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:48:01.607281 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:48:01.609107 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:48:01.609874 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:48:01.609881 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:48:01.609884 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:48:01.610045 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:48:01.610470 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:48:01.610477 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:48:01.610480 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:48:01.610762 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:48:01.610904 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0812 09:48:01.611464 (+ 560us) server_negotiation.cc:299] Negotiation successful
0812 09:48:01.611630 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260812 09:48:01.612624 5888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:01.597642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:42637 (local address 127.13.99.2:52753)
0812 09:48:01.598130 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0812 09:48:01.598173 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0812 09:48:01.599504 (+ 1331us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:48:01.600188 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:48:01.600196 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:48:01.600582 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:48:01.601068 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:48:01.601080 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:48:01.602164 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:48:01.602170 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:48:01.603696 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:48:01.603712 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:48:01.604232 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:48:01.604239 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:48:01.604514 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:48:01.605172 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:48:01.605197 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:48:01.607040 (+ 1843us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:48:01.609280 (+ 2240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:48:01.609289 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:48:01.609306 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:48:01.609733 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:48:01.610180 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:48:01.610183 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:48:01.610185 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:48:01.610364 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:48:01.610921 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:48:01.610927 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:48:01.611180 (+ 253us) client_negotiation.cc:769] Sending connection context
0812 09:48:01.611363 (+ 183us) client_negotiation.cc:240] Negotiation successful
0812 09:48:01.611631 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":120,"threads_started":1}
I20260812 09:48:01.613922 5881 heartbeater.cc:347] Connected to a master server at 127.13.99.62:42637
I20260812 09:48:01.614168 5881 heartbeater.cc:464] Registering TS with master...
I20260812 09:48:01.614689 5881 heartbeater.cc:511] Master 127.13.99.62:42637 requested a full tablet report, sending...
I20260812 09:48:01.615842 5242 ts_manager.cc:194] Registered new tserver with Master: 0b12969c05f24d038e309045b035f69e (127.13.99.2:39701)
I20260812 09:48:01.617548 5242 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.99.2@KRBTEST.COM'} at 127.13.99.2:52753
W20260812 09:48:01.721463 5890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:48:01.721704 5890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:48:01.721781 5890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:48:01.725781 5890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:48:01.725835 5890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:48:01.725939 5890 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.99.3
I20260812 09:48:01.731071 5890 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:33335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.3.keytab
--principal=kudu/127.13.99.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.99.3:32841
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.99.3
--webserver_port=43579
--tserver_master_addrs=127.13.99.62:42637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5890
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.99.3
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:48:01.732070 5890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:48:01.733139 5890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:48:01.742549 5902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:48:01.742667 5890 server_base.cc:1034] running on GCE node
W20260812 09:48:01.742569 5898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:48:01.742594 5899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:48:01.743216 5890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:48:01.743880 5890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:48:01.745083 5890 hybrid_clock.cc:648] HybridClock initialized: now 1786528081745071 us; error 29 us; skew 500 ppm
Aug 12 09:48:01 dist-test-slave-jq4t krb5kdc[4086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528081, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:48:01.747975 5890 init.cc:377] Logged in from keytab as kudu/127.13.99.3@KRBTEST.COM (short username kudu)
I20260812 09:48:01.748988 5890 webserver.cc:460] Webserver started at http://127.13.99.3:43579/ using document root <none> and password file <none>
I20260812 09:48:01.749577 5890 fs_manager.cc:359] Metadata directory not provided
I20260812 09:48:01.749688 5890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:48:01.753846 5890 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 09:48:01.756018 5912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:48:01.757370 5890 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 09:48:01.757513 5890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-2/wal
uuid: "5cdb8b929df74926b65c64638cbbd3f5"
format_stamp: "Formatted at 2026-08-12 09:48:00 on dist-test-slave-jq4t"
server_key: "db9007ea05f29349965f91af0a28b62e"
server_key_iv: "aada7159e0a0a68d425fc6216d560d35"
server_key_version: "encryptionkey@0"
I20260812 09:48:01.758072 5890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:48:01.781270 5890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:48:01.784714 5890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:48:01.784927 5890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:48:01.785548 5890 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 09:48:01.786504 5890 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:48:01.786586 5890 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 09:48:01.786661 5890 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:48:01.786697 5890 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 09:48:01.795708 5890 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.3:32841
I20260812 09:48:01.795745 6151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.3:32841 every 8 connection(s)
I20260812 09:48:01.796718 5890 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-2/data/info.pb
I20260812 09:48:01.799243 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 5890
Aug 12 09:48:01 dist-test-slave-jq4t krb5kdc[4086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528081, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.3@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
I20260812 09:48:01.809191 6156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:01.799923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49484 (local address 127.13.99.62:42637)
0812 09:48:01.800257 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0812 09:48:01.800260 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 09:48:01.800382 (+ 122us) server_negotiation.cc:407] Connection header received
0812 09:48:01.800570 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:48:01.800573 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:48:01.800618 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:48:01.800701 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:48:01.801632 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:48:01.802247 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:48:01.803428 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:48:01.803788 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:48:01.803873 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:48:01.805647 (+ 1774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:48:01.805651 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:48:01.805654 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:48:01.805682 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:48:01.807379 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:48:01.807857 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:48:01.807863 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:48:01.807868 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:48:01.807990 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:48:01.808289 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:48:01.808292 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:48:01.808294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:48:01.808530 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:48:01.808733 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0812 09:48:01.808883 (+ 150us) server_negotiation.cc:299] Negotiation successful
0812 09:48:01.809007 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":275,"thread_start_us":92,"threads_started":1}
I20260812 09:48:01.812664 5626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:01.798427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.3:58263 (local address 127.13.99.62:42637)
0812 09:48:01.798595 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0812 09:48:01.798600 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:48:01.799577 (+ 977us) server_negotiation.cc:407] Connection header received
0812 09:48:01.801543 (+ 1966us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:48:01.801545 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:48:01.801605 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:48:01.801691 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:48:01.803000 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:48:01.803684 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:48:01.805500 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:48:01.805817 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:48:01.805953 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:48:01.808925 (+ 2972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:48:01.808928 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:48:01.808931 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:48:01.808963 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:48:01.810753 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:48:01.811351 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:48:01.811354 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:48:01.811355 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:48:01.811434 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:48:01.811733 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:48:01.811735 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:48:01.811736 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:48:01.811886 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:48:01.811960 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0812 09:48:01.812400 (+ 440us) server_negotiation.cc:299] Negotiation successful
0812 09:48:01.812549 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":67}
I20260812 09:48:01.813470 6155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:01.798806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:42637 (local address 127.13.99.3:58263)
0812 09:48:01.799405 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0812 09:48:01.799472 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0812 09:48:01.801304 (+ 1832us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:48:01.801866 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:48:01.801873 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:48:01.802201 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:48:01.802846 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:48:01.802867 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:48:01.803816 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:48:01.803819 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:48:01.805290 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:48:01.805302 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:48:01.806019 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:48:01.806025 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:48:01.806251 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:48:01.807009 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:48:01.807024 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:48:01.808739 (+ 1715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:48:01.810870 (+ 2131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:48:01.810875 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:48:01.810887 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:48:01.811244 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:48:01.811513 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:48:01.811516 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:48:01.811517 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:48:01.811642 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:48:01.811975 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:48:01.811980 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:48:01.812194 (+ 214us) client_negotiation.cc:769] Sending connection context
0812 09:48:01.812335 (+ 141us) client_negotiation.cc:240] Negotiation successful
0812 09:48:01.812645 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":200,"threads_started":1}
I20260812 09:48:01.814640 6153 heartbeater.cc:347] Connected to a master server at 127.13.99.62:42637
I20260812 09:48:01.814879 6153 heartbeater.cc:464] Registering TS with master...
I20260812 09:48:01.815362 6153 heartbeater.cc:511] Master 127.13.99.62:42637 requested a full tablet report, sending...
I20260812 09:48:01.816388 5242 ts_manager.cc:194] Registered new tserver with Master: 5cdb8b929df74926b65c64638cbbd3f5 (127.13.99.3:32841)
I20260812 09:48:01.816993 5242 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.99.3@KRBTEST.COM'} at 127.13.99.3:58263
I20260812 09:48:01.822767 13708 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 09:48:01.826692 5626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:01.823349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49486 (local address 127.13.99.62:42637)
0812 09:48:01.823464 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0812 09:48:01.823467 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 09:48:01.823663 (+ 196us) server_negotiation.cc:407] Connection header received
0812 09:48:01.823785 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:48:01.823787 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:48:01.823870 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:48:01.823960 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 09:48:01.824140 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:48:01.824671 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:48:01.825477 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:48:01.825678 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:48:01.825764 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:48:01.826073 (+ 309us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 09:48:01.826410 (+ 337us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 09:48:01.826503 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0812 09:48:01.826561 (+ 58us) server_negotiation.cc:299] Negotiation successful
0812 09:48:01.826603 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260812 09:48:01.827281 5242 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:49486:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 09:48:01.830556 5242 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 09:48:01.840036 5468 tablet_service.cc:1467] Processing CreateTablet for tablet 73421e2debc3482a88a8900d7c169268 (DEFAULT_TABLE table=test-table [id=fe09e69cdd0241cd9662f0e2e1e4dee6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 09:48:01.840816 5468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73421e2debc3482a88a8900d7c169268. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:48:01.843715 5630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:01.838600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.2:39701 (local address 127.0.0.1:59164)
0812 09:48:01.838784 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0812 09:48:01.838802 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 09:48:01.838991 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:48:01.839772 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:48:01.839775 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:48:01.839789 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:48:01.839940 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:48:01.839947 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:48:01.841465 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:48:01.841470 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:48:01.842267 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:48:01.842276 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:48:01.843300 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:48:01.843306 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:48:01.843408 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:48:01.843468 (+ 60us) client_negotiation.cc:769] Sending connection context
0812 09:48:01.843516 (+ 48us) client_negotiation.cc:240] Negotiation successful
0812 09:48:01.843583 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104}
I20260812 09:48:01.843813 5403 tablet_bootstrap.cc:492] T 73421e2debc3482a88a8900d7c169268 P a808eee3e1c14f9488cefe0d16ccc66c: Bootstrap starting.
I20260812 09:48:01.844523 6171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:01.838784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.3:32841 (local address 127.0.0.1:34680)
0812 09:48:01.839689 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0812 09:48:01.839706 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 09:48:01.839782 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:48:01.840420 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:48:01.840426 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:48:01.840449 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:48:01.840561 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:48:01.840567 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:48:01.841898 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:48:01.841901 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:48:01.843005 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:48:01.843016 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:48:01.844096 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:48:01.844099 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:48:01.844185 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:48:01.844206 (+ 21us) client_negotiation.cc:769] Sending connection context
0812 09:48:01.844271 (+ 65us) client_negotiation.cc:240] Negotiation successful
0812 09:48:01.844338 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":822,"mutex_wait_us":2,"thread_start_us":112,"threads_started":1}
I20260812 09:48:01.844537 6172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:01.839219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34680 (local address 127.13.99.3:32841)
0812 09:48:01.839777 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0812 09:48:01.839789 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 09:48:01.839813 (+ 24us) server_negotiation.cc:407] Connection header received
0812 09:48:01.839887 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:48:01.839892 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:48:01.840091 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:48:01.840250 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:48:01.840702 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:48:01.841743 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:48:01.843151 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:48:01.843972 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:48:01.844095 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:48:01.844246 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0812 09:48:01.844302 (+ 56us) server_negotiation.cc:299] Negotiation successful
0812 09:48:01.844384 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":160,"threads_started":1}
I20260812 09:48:01.844604 6167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:01.838771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59164 (local address 127.13.99.2:39701)
0812 09:48:01.839187 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0812 09:48:01.839201 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 09:48:01.839226 (+ 25us) server_negotiation.cc:407] Connection header received
0812 09:48:01.839320 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:48:01.839327 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:48:01.839584 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:48:01.840111 (+ 527us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:48:01.840350 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:48:01.841328 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:48:01.842398 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:48:01.843162 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:48:01.844143 (+ 981us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:48:01.844285 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0812 09:48:01.844335 (+ 50us) server_negotiation.cc:299] Negotiation successful
0812 09:48:01.844418 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":124,"threads_started":1}
I20260812 09:48:01.844718 5403 tablet_bootstrap.cc:654] T 73421e2debc3482a88a8900d7c169268 P a808eee3e1c14f9488cefe0d16ccc66c: Neither blocks nor log segments found. Creating new log.
I20260812 09:48:01.846338 5403 tablet_bootstrap.cc:492] T 73421e2debc3482a88a8900d7c169268 P a808eee3e1c14f9488cefe0d16ccc66c: No bootstrap required, opened a new log
I20260812 09:48:01.846422 5403 ts_tablet_manager.cc:1397] T 73421e2debc3482a88a8900d7c169268 P a808eee3e1c14f9488cefe0d16ccc66c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 09:48:01.846858 5403 raft_consensus.cc:348] T 73421e2debc3482a88a8900d7c169268 P a808eee3e1c14f9488cefe0d16ccc66c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b12969c05f24d038e309045b035f69e" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 39701 } } peers { permanent_uuid: "5cdb8b929df74926b65c64638cbbd3f5" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 32841 } } peers { permanent_uuid: "a808eee3e1c14f9488cefe0d16ccc66c" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 43137 } }
I20260812 09:48:01.846977 5403 raft_consensus.cc:374] T 73421e2debc3482a88a8900d7c169268 P a808eee3e1c14f9488cefe0d16ccc66c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:48:01.847018 5403 raft_consensus.cc:729] T 73421e2debc3482a88a8900d7c169268 P a808eee3e1c14f9488cefe0d16ccc66c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a808eee3e1c14f9488cefe0d16ccc66c, State: Initialized, Role: FOLLOWER
I20260812 09:48:01.847136 5403 consensus_queue.cc:260] T 73421e2debc3482a88a8900d7c169268 P a808eee3e1c14f9488cefe0d16ccc66c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b12969c05f24d038e309045b035f69e" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 39701 } } peers { permanent_uuid: "5cdb8b929df74926b65c64638cbbd3f5" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 32841 } } peers { permanent_uuid: "a808eee3e1c14f9488cefe0d16ccc66c" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 43137 } }
I20260812 09:48:01.847381 5403 ts_tablet_manager.cc:1428] T 73421e2debc3482a88a8900d7c169268 P a808eee3e1c14f9488cefe0d16ccc66c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260812 09:48:01.847597 5994 tablet_service.cc:1467] Processing CreateTablet for tablet 73421e2debc3482a88a8900d7c169268 (DEFAULT_TABLE table=test-table [id=fe09e69cdd0241cd9662f0e2e1e4dee6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 09:48:01.847605 5756 tablet_service.cc:1467] Processing CreateTablet for tablet 73421e2debc3482a88a8900d7c169268 (DEFAULT_TABLE table=test-table [id=fe09e69cdd0241cd9662f0e2e1e4dee6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 09:48:01.848675 5994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73421e2debc3482a88a8900d7c169268. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:48:01.848696 5756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73421e2debc3482a88a8900d7c169268. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:48:01.855609 6174 tablet_bootstrap.cc:492] T 73421e2debc3482a88a8900d7c169268 P 5cdb8b929df74926b65c64638cbbd3f5: Bootstrap starting.
I20260812 09:48:01.855715 5629 raft_consensus.cc:482] T 73421e2debc3482a88a8900d7c169268 P a808eee3e1c14f9488cefe0d16ccc66c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 09:48:01.855867 5629 raft_consensus.cc:504] T 73421e2debc3482a88a8900d7c169268 P a808eee3e1c14f9488cefe0d16ccc66c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b12969c05f24d038e309045b035f69e" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 39701 } } peers { permanent_uuid: "5cdb8b929df74926b65c64638cbbd3f5" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 32841 } } peers { permanent_uuid: "a808eee3e1c14f9488cefe0d16ccc66c" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 43137 } }
I20260812 09:48:01.856506 6176 tablet_bootstrap.cc:492] T 73421e2debc3482a88a8900d7c169268 P 0b12969c05f24d038e309045b035f69e: Bootstrap starting.
I20260812 09:48:01.856979 5629 leader_election.cc:288] T 73421e2debc3482a88a8900d7c169268 P a808eee3e1c14f9488cefe0d16ccc66c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b12969c05f24d038e309045b035f69e (127.13.99.2:39701), 5cdb8b929df74926b65c64638cbbd3f5 (127.13.99.3:32841)
I20260812 09:48:01.858634 6174 tablet_bootstrap.cc:654] T 73421e2debc3482a88a8900d7c169268 P 5cdb8b929df74926b65c64638cbbd3f5: Neither blocks nor log segments found. Creating new log.
I20260812 09:48:01.859140 6176 tablet_bootstrap.cc:654] T 73421e2debc3482a88a8900d7c169268 P 0b12969c05f24d038e309045b035f69e: Neither blocks nor log segments found. Creating new log.
I20260812 09:48:01.859997 6176 log.cc:824] T 73421e2debc3482a88a8900d7c169268 P 0b12969c05f24d038e309045b035f69e: Log is configured to *not* fsync() on all Append() calls
I20260812 09:48:01.859995 6174 log.cc:824] T 73421e2debc3482a88a8900d7c169268 P 5cdb8b929df74926b65c64638cbbd3f5: Log is configured to *not* fsync() on all Append() calls
I20260812 09:48:01.860952 5624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:01.857130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.2:39701 (local address 127.13.99.1:59845)
0812 09:48:01.857543 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0812 09:48:01.857558 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 09:48:01.857656 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:48:01.857948 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:48:01.857951 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:48:01.857969 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:48:01.858086 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:48:01.858091 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:48:01.858817 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:48:01.858820 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:48:01.859653 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:48:01.859677 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:48:01.860376 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:48:01.860379 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:48:01.860542 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:48:01.860604 (+ 62us) client_negotiation.cc:769] Sending connection context
0812 09:48:01.860687 (+ 83us) client_negotiation.cc:240] Negotiation successful
0812 09:48:01.860785 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349}
I20260812 09:48:01.861289 6167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:01.857339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.1:59845 (local address 127.13.99.2:39701)
0812 09:48:01.857466 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0812 09:48:01.857472 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 09:48:01.857607 (+ 135us) server_negotiation.cc:407] Connection header received
0812 09:48:01.857799 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:48:01.857803 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:48:01.857849 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:48:01.857924 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:48:01.858192 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:48:01.858660 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:48:01.859785 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:48:01.860239 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:48:01.860377 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:48:01.860435 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0812 09:48:01.861125 (+ 690us) server_negotiation.cc:299] Negotiation successful
0812 09:48:01.861191 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260812 09:48:01.862160 6174 tablet_bootstrap.cc:492] T 73421e2debc3482a88a8900d7c169268 P 5cdb8b929df74926b65c64638cbbd3f5: No bootstrap required, opened a new log
I20260812 09:48:01.862210 6176 tablet_bootstrap.cc:492] T 73421e2debc3482a88a8900d7c169268 P 0b12969c05f24d038e309045b035f69e: No bootstrap required, opened a new log
I20260812 09:48:01.862353 6174 ts_tablet_manager.cc:1397] T 73421e2debc3482a88a8900d7c169268 P 5cdb8b929df74926b65c64638cbbd3f5: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260812 09:48:01.862377 6176 ts_tablet_manager.cc:1397] T 73421e2debc3482a88a8900d7c169268 P 0b12969c05f24d038e309045b035f69e: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260812 09:48:01.861954 5789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73421e2debc3482a88a8900d7c169268" candidate_uuid: "a808eee3e1c14f9488cefe0d16ccc66c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b12969c05f24d038e309045b035f69e" is_pre_election: true
W20260812 09:48:01.863993 5393 leader_election.cc:341] T 73421e2debc3482a88a8900d7c169268 P a808eee3e1c14f9488cefe0d16ccc66c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b12969c05f24d038e309045b035f69e (127.13.99.2:39701): Illegal state: must be running to vote when last-logged opid is not known
I20260812 09:48:01.864831 6177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:01.857252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.3:32841 (local address 127.13.99.1:53037)
0812 09:48:01.860572 (+ 3320us) negotiation.cc:107] Waiting for socket to connect
0812 09:48:01.860590 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 09:48:01.860692 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:48:01.861078 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:48:01.861081 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:48:01.861098 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:48:01.861197 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:48:01.861203 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:48:01.862147 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:48:01.862150 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:48:01.863446 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:48:01.863457 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:48:01.864397 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:48:01.864401 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:48:01.864534 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:48:01.864558 (+ 24us) client_negotiation.cc:769] Sending connection context
0812 09:48:01.864630 (+ 72us) client_negotiation.cc:240] Negotiation successful
0812 09:48:01.864694 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3231,"thread_start_us":109,"threads_started":1}
I20260812 09:48:01.865167 6172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:01.857338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.1:53037 (local address 127.13.99.3:32841)
0812 09:48:01.857563 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0812 09:48:01.857567 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:48:01.860780 (+ 3213us) server_negotiation.cc:407] Connection header received
0812 09:48:01.860829 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:48:01.860834 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:48:01.860889 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:48:01.860976 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:48:01.861316 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:48:01.862014 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:48:01.863587 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:48:01.864170 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:48:01.864278 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:48:01.864333 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0812 09:48:01.864994 (+ 661us) server_negotiation.cc:299] Negotiation successful
0812 09:48:01.865064 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155}
I20260812 09:48:01.865648 6032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73421e2debc3482a88a8900d7c169268" candidate_uuid: "a808eee3e1c14f9488cefe0d16ccc66c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cdb8b929df74926b65c64638cbbd3f5" is_pre_election: true
I20260812 09:48:01.866086 6174 raft_consensus.cc:348] T 73421e2debc3482a88a8900d7c169268 P 5cdb8b929df74926b65c64638cbbd3f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b12969c05f24d038e309045b035f69e" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 39701 } } peers { permanent_uuid: "5cdb8b929df74926b65c64638cbbd3f5" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 32841 } } peers { permanent_uuid: "a808eee3e1c14f9488cefe0d16ccc66c" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 43137 } }
I20260812 09:48:01.865365 6176 raft_consensus.cc:348] T 73421e2debc3482a88a8900d7c169268 P 0b12969c05f24d038e309045b035f69e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b12969c05f24d038e309045b035f69e" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 39701 } } peers { permanent_uuid: "5cdb8b929df74926b65c64638cbbd3f5" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 32841 } } peers { permanent_uuid: "a808eee3e1c14f9488cefe0d16ccc66c" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 43137 } }
W20260812 09:48:01.866321 5394 leader_election.cc:341] T 73421e2debc3482a88a8900d7c169268 P a808eee3e1c14f9488cefe0d16ccc66c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5cdb8b929df74926b65c64638cbbd3f5 (127.13.99.3:32841): Illegal state: must be running to vote when last-logged opid is not known
I20260812 09:48:01.866381 6176 raft_consensus.cc:374] T 73421e2debc3482a88a8900d7c169268 P 0b12969c05f24d038e309045b035f69e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:48:01.866369 6174 raft_consensus.cc:374] T 73421e2debc3482a88a8900d7c169268 P 5cdb8b929df74926b65c64638cbbd3f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:48:01.866405 5394 leader_election.cc:302] T 73421e2debc3482a88a8900d7c169268 P a808eee3e1c14f9488cefe0d16ccc66c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a808eee3e1c14f9488cefe0d16ccc66c; no voters: 0b12969c05f24d038e309045b035f69e, 5cdb8b929df74926b65c64638cbbd3f5
I20260812 09:48:01.866443 6174 raft_consensus.cc:729] T 73421e2debc3482a88a8900d7c169268 P 5cdb8b929df74926b65c64638cbbd3f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cdb8b929df74926b65c64638cbbd3f5, State: Initialized, Role: FOLLOWER
I20260812 09:48:01.866442 6176 raft_consensus.cc:729] T 73421e2debc3482a88a8900d7c169268 P 0b12969c05f24d038e309045b035f69e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b12969c05f24d038e309045b035f69e, State: Initialized, Role: FOLLOWER
I20260812 09:48:01.866683 5629 raft_consensus.cc:2726] T 73421e2debc3482a88a8900d7c169268 P a808eee3e1c14f9488cefe0d16ccc66c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 09:48:01.866938 6174 consensus_queue.cc:260] T 73421e2debc3482a88a8900d7c169268 P 5cdb8b929df74926b65c64638cbbd3f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b12969c05f24d038e309045b035f69e" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 39701 } } peers { permanent_uuid: "5cdb8b929df74926b65c64638cbbd3f5" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 32841 } } peers { permanent_uuid: "a808eee3e1c14f9488cefe0d16ccc66c" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 43137 } }
I20260812 09:48:01.866950 6176 consensus_queue.cc:260] T 73421e2debc3482a88a8900d7c169268 P 0b12969c05f24d038e309045b035f69e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b12969c05f24d038e309045b035f69e" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 39701 } } peers { permanent_uuid: "5cdb8b929df74926b65c64638cbbd3f5" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 32841 } } peers { permanent_uuid: "a808eee3e1c14f9488cefe0d16ccc66c" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 43137 } }
I20260812 09:48:01.867556 5881 heartbeater.cc:503] Master 127.13.99.62:42637 was elected leader, sending a full tablet report...
I20260812 09:48:01.867556 6153 heartbeater.cc:503] Master 127.13.99.62:42637 was elected leader, sending a full tablet report...
I20260812 09:48:01.867937 6176 ts_tablet_manager.cc:1428] T 73421e2debc3482a88a8900d7c169268 P 0b12969c05f24d038e309045b035f69e: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260812 09:48:01.868923 6174 ts_tablet_manager.cc:1428] T 73421e2debc3482a88a8900d7c169268 P 5cdb8b929df74926b65c64638cbbd3f5: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
W20260812 09:48:01.903187 5623 tablet.cc:2369] T 73421e2debc3482a88a8900d7c169268 P a808eee3e1c14f9488cefe0d16ccc66c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 09:48:02.027707 6184 raft_consensus.cc:482] T 73421e2debc3482a88a8900d7c169268 P 0b12969c05f24d038e309045b035f69e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 09:48:02.028025 6184 raft_consensus.cc:504] T 73421e2debc3482a88a8900d7c169268 P 0b12969c05f24d038e309045b035f69e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b12969c05f24d038e309045b035f69e" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 39701 } } peers { permanent_uuid: "5cdb8b929df74926b65c64638cbbd3f5" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 32841 } } peers { permanent_uuid: "a808eee3e1c14f9488cefe0d16ccc66c" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 43137 } }
I20260812 09:48:02.029387 6184 leader_election.cc:288] T 73421e2debc3482a88a8900d7c169268 P 0b12969c05f24d038e309045b035f69e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5cdb8b929df74926b65c64638cbbd3f5 (127.13.99.3:32841), a808eee3e1c14f9488cefe0d16ccc66c (127.13.99.1:43137)
I20260812 09:48:02.033299 6172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:02.029729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.2:55133 (local address 127.13.99.3:32841)
0812 09:48:02.029897 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0812 09:48:02.029903 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 09:48:02.029923 (+ 20us) server_negotiation.cc:407] Connection header received
0812 09:48:02.030010 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:48:02.030013 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:48:02.030074 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:48:02.030157 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:48:02.030515 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:48:02.031110 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:48:02.032208 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:48:02.032713 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:48:02.032795 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:48:02.032855 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0812 09:48:02.033092 (+ 237us) server_negotiation.cc:299] Negotiation successful
0812 09:48:02.033162 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260812 09:48:02.033288 5888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:02.029614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.3:32841 (local address 127.13.99.2:55133)
0812 09:48:02.029798 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0812 09:48:02.029816 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 09:48:02.029907 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:48:02.030214 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:48:02.030217 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:48:02.030239 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:48:02.030391 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:48:02.030399 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:48:02.031233 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:48:02.031237 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:48:02.032074 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:48:02.032083 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:48:02.032831 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:48:02.032835 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:48:02.032955 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:48:02.032986 (+ 31us) client_negotiation.cc:769] Sending connection context
0812 09:48:02.033063 (+ 77us) client_negotiation.cc:240] Negotiation successful
0812 09:48:02.033134 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"mutex_wait_us":1}
I20260812 09:48:02.033479 6186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:02.029635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.1:43137 (local address 127.13.99.2:47099)
0812 09:48:02.030018 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0812 09:48:02.030032 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 09:48:02.030126 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:48:02.030405 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:48:02.030407 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:48:02.030422 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:48:02.030562 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:48:02.030569 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:48:02.031374 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:48:02.031376 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:48:02.032185 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:48:02.032198 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:48:02.033004 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:48:02.033007 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:48:02.033130 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:48:02.033152 (+ 22us) client_negotiation.cc:769] Sending connection context
0812 09:48:02.033230 (+ 78us) client_negotiation.cc:240] Negotiation successful
0812 09:48:02.033295 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"mutex_wait_us":1,"thread_start_us":119,"threads_started":1}
I20260812 09:48:02.033852 6190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:02.029748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.2:47099 (local address 127.13.99.1:43137)
0812 09:48:02.030070 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0812 09:48:02.030073 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 09:48:02.030086 (+ 13us) server_negotiation.cc:407] Connection header received
0812 09:48:02.030241 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:48:02.030244 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:48:02.030290 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:48:02.030386 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:48:02.030678 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:48:02.031237 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:48:02.032345 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:48:02.032881 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:48:02.032987 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:48:02.033141 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0812 09:48:02.033241 (+ 100us) server_negotiation.cc:299] Negotiation successful
0812 09:48:02.033313 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":93,"threads_started":1}
I20260812 09:48:02.033885 6032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73421e2debc3482a88a8900d7c169268" candidate_uuid: "0b12969c05f24d038e309045b035f69e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cdb8b929df74926b65c64638cbbd3f5" is_pre_election: true
I20260812 09:48:02.034214 6032 raft_consensus.cc:2454] T 73421e2debc3482a88a8900d7c169268 P 5cdb8b929df74926b65c64638cbbd3f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b12969c05f24d038e309045b035f69e in term 0.
I20260812 09:48:02.034334 5531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73421e2debc3482a88a8900d7c169268" candidate_uuid: "0b12969c05f24d038e309045b035f69e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a808eee3e1c14f9488cefe0d16ccc66c" is_pre_election: true
I20260812 09:48:02.034586 5531 raft_consensus.cc:2454] T 73421e2debc3482a88a8900d7c169268 P a808eee3e1c14f9488cefe0d16ccc66c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b12969c05f24d038e309045b035f69e in term 0.
I20260812 09:48:02.034608 5673 leader_election.cc:302] T 73421e2debc3482a88a8900d7c169268 P 0b12969c05f24d038e309045b035f69e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b12969c05f24d038e309045b035f69e, 5cdb8b929df74926b65c64638cbbd3f5; no voters:
I20260812 09:48:02.034927 6184 raft_consensus.cc:2781] T 73421e2debc3482a88a8900d7c169268 P 0b12969c05f24d038e309045b035f69e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 09:48:02.035046 6184 raft_consensus.cc:482] T 73421e2debc3482a88a8900d7c169268 P 0b12969c05f24d038e309045b035f69e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 09:48:02.035109 6184 raft_consensus.cc:3037] T 73421e2debc3482a88a8900d7c169268 P 0b12969c05f24d038e309045b035f69e [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:48:02.036242 6184 raft_consensus.cc:504] T 73421e2debc3482a88a8900d7c169268 P 0b12969c05f24d038e309045b035f69e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b12969c05f24d038e309045b035f69e" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 39701 } } peers { permanent_uuid: "5cdb8b929df74926b65c64638cbbd3f5" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 32841 } } peers { permanent_uuid: "a808eee3e1c14f9488cefe0d16ccc66c" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 43137 } }
I20260812 09:48:02.036618 6184 leader_election.cc:288] T 73421e2debc3482a88a8900d7c169268 P 0b12969c05f24d038e309045b035f69e [CANDIDATE]: Term 1 election: Requested vote from peers 5cdb8b929df74926b65c64638cbbd3f5 (127.13.99.3:32841), a808eee3e1c14f9488cefe0d16ccc66c (127.13.99.1:43137)
I20260812 09:48:02.037004 5531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73421e2debc3482a88a8900d7c169268" candidate_uuid: "0b12969c05f24d038e309045b035f69e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a808eee3e1c14f9488cefe0d16ccc66c"
I20260812 09:48:02.037056 6032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73421e2debc3482a88a8900d7c169268" candidate_uuid: "0b12969c05f24d038e309045b035f69e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cdb8b929df74926b65c64638cbbd3f5"
I20260812 09:48:02.037146 5531 raft_consensus.cc:3037] T 73421e2debc3482a88a8900d7c169268 P a808eee3e1c14f9488cefe0d16ccc66c [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:48:02.037163 6032 raft_consensus.cc:3037] T 73421e2debc3482a88a8900d7c169268 P 5cdb8b929df74926b65c64638cbbd3f5 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:48:02.038359 5531 raft_consensus.cc:2454] T 73421e2debc3482a88a8900d7c169268 P a808eee3e1c14f9488cefe0d16ccc66c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b12969c05f24d038e309045b035f69e in term 1.
I20260812 09:48:02.038359 6032 raft_consensus.cc:2454] T 73421e2debc3482a88a8900d7c169268 P 5cdb8b929df74926b65c64638cbbd3f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b12969c05f24d038e309045b035f69e in term 1.
I20260812 09:48:02.038658 5669 leader_election.cc:302] T 73421e2debc3482a88a8900d7c169268 P 0b12969c05f24d038e309045b035f69e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b12969c05f24d038e309045b035f69e, a808eee3e1c14f9488cefe0d16ccc66c; no voters:
I20260812 09:48:02.038856 6184 raft_consensus.cc:2781] T 73421e2debc3482a88a8900d7c169268 P 0b12969c05f24d038e309045b035f69e [term 1 FOLLOWER]: Leader election won for term 1
I20260812 09:48:02.039050 6184 raft_consensus.cc:686] T 73421e2debc3482a88a8900d7c169268 P 0b12969c05f24d038e309045b035f69e [term 1 LEADER]: Becoming Leader. State: Replica: 0b12969c05f24d038e309045b035f69e, State: Running, Role: LEADER
I20260812 09:48:02.039386 6184 consensus_queue.cc:237] T 73421e2debc3482a88a8900d7c169268 P 0b12969c05f24d038e309045b035f69e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b12969c05f24d038e309045b035f69e" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 39701 } } peers { permanent_uuid: "5cdb8b929df74926b65c64638cbbd3f5" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 32841 } } peers { permanent_uuid: "a808eee3e1c14f9488cefe0d16ccc66c" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 43137 } }
I20260812 09:48:02.042659 5242 catalog_manager.cc:5495] T 73421e2debc3482a88a8900d7c169268 P 0b12969c05f24d038e309045b035f69e reported cstate change: term changed from 0 to 1, leader changed from <none> to 0b12969c05f24d038e309045b035f69e (127.13.99.2). New cstate: current_term: 1 leader_uuid: "0b12969c05f24d038e309045b035f69e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b12969c05f24d038e309045b035f69e" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 39701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5cdb8b929df74926b65c64638cbbd3f5" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 32841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a808eee3e1c14f9488cefe0d16ccc66c" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 43137 } health_report { overall_health: UNKNOWN } } }
W20260812 09:48:02.047610 6154 tablet.cc:2369] T 73421e2debc3482a88a8900d7c169268 P 5cdb8b929df74926b65c64638cbbd3f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 09:48:02.082831 6167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:02.079492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59170 (local address 127.13.99.2:39701)
0812 09:48:02.079635 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0812 09:48:02.079638 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 09:48:02.079653 (+ 15us) server_negotiation.cc:407] Connection header received
0812 09:48:02.079808 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:48:02.079811 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:48:02.079870 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:48:02.079963 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 09:48:02.080183 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:48:02.080701 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:48:02.081474 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:48:02.081687 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:48:02.081761 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:48:02.082015 (+ 254us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 09:48:02.082423 (+ 408us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 09:48:02.082513 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0812 09:48:02.082662 (+ 149us) server_negotiation.cc:299] Negotiation successful
0812 09:48:02.082710 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260812 09:48:02.088860 6032 raft_consensus.cc:1261] T 73421e2debc3482a88a8900d7c169268 P 5cdb8b929df74926b65c64638cbbd3f5 [term 1 FOLLOWER]: Refusing update from remote peer 0b12969c05f24d038e309045b035f69e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 09:48:02.088888 5531 raft_consensus.cc:1261] T 73421e2debc3482a88a8900d7c169268 P a808eee3e1c14f9488cefe0d16ccc66c [term 1 FOLLOWER]: Refusing update from remote peer 0b12969c05f24d038e309045b035f69e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 09:48:02.089612 6184 consensus_queue.cc:1035] T 73421e2debc3482a88a8900d7c169268 P 0b12969c05f24d038e309045b035f69e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a808eee3e1c14f9488cefe0d16ccc66c" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 43137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 09:48:02.089812 6191 consensus_queue.cc:1035] T 73421e2debc3482a88a8900d7c169268 P 0b12969c05f24d038e309045b035f69e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cdb8b929df74926b65c64638cbbd3f5" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 32841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 09:48:02.096880 6203 mvcc.cc:205] Tried to move back new op lower bound from 7317619024227753984 to 7317619024036196352. Current Snapshot: MvccSnapshot[applied={T|T < 7317619024227753984}]
I20260812 09:48:02.105105 5239 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:49486:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 09:48:02.105242 5239 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:49486:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 09:48:02.107656 5239 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 83d811857241469db629144445c55d66: Sending DeleteTablet for 3 replicas of tablet 73421e2debc3482a88a8900d7c169268
I20260812 09:48:02.108337 5468 tablet_service.cc:1514] Processing DeleteTablet for tablet 73421e2debc3482a88a8900d7c169268 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 09:48:02 UTC) from {username='kudu', principal='kudu/127.13.99.62@KRBTEST.COM'} at 127.0.0.1:45920
I20260812 09:48:02.108351 5994 tablet_service.cc:1514] Processing DeleteTablet for tablet 73421e2debc3482a88a8900d7c169268 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 09:48:02 UTC) from {username='kudu', principal='kudu/127.13.99.62@KRBTEST.COM'} at 127.0.0.1:34680
I20260812 09:48:02.108572 5756 tablet_service.cc:1514] Processing DeleteTablet for tablet 73421e2debc3482a88a8900d7c169268 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 09:48:02 UTC) from {username='kudu', principal='kudu/127.13.99.62@KRBTEST.COM'} at 127.0.0.1:59164
I20260812 09:48:02.108794 6225 tablet_replica.cc:330] T 73421e2debc3482a88a8900d7c169268 P 5cdb8b929df74926b65c64638cbbd3f5: stopping tablet replica
I20260812 09:48:02.108939 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 5322
I20260812 09:48:02.109113 6225 raft_consensus.cc:2229] T 73421e2debc3482a88a8900d7c169268 P 5cdb8b929df74926b65c64638cbbd3f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 09:48:02.109175 6226 tablet_replica.cc:330] T 73421e2debc3482a88a8900d7c169268 P 0b12969c05f24d038e309045b035f69e: stopping tablet replica
I20260812 09:48:02.109411 6225 raft_consensus.cc:2258] T 73421e2debc3482a88a8900d7c169268 P 5cdb8b929df74926b65c64638cbbd3f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 09:48:02.109633 6226 raft_consensus.cc:2229] T 73421e2debc3482a88a8900d7c169268 P 0b12969c05f24d038e309045b035f69e [term 1 LEADER]: Raft consensus shutting down.
I20260812 09:48:02.109977 6226 raft_consensus.cc:2258] T 73421e2debc3482a88a8900d7c169268 P 0b12969c05f24d038e309045b035f69e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 09:48:02.114537 6226 ts_tablet_manager.cc:1905] T 73421e2debc3482a88a8900d7c169268 P 0b12969c05f24d038e309045b035f69e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 09:48:02.117542 6226 ts_tablet_manager.cc:1918] T 73421e2debc3482a88a8900d7c169268 P 0b12969c05f24d038e309045b035f69e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 09:48:02.117645 6226 log.cc:1196] T 73421e2debc3482a88a8900d7c169268 P 0b12969c05f24d038e309045b035f69e: Deleting WAL directory at /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-1/wal/wals/73421e2debc3482a88a8900d7c169268
I20260812 09:48:02.118065 6226 ts_tablet_manager.cc:1939] T 73421e2debc3482a88a8900d7c169268 P 0b12969c05f24d038e309045b035f69e: Deleting consensus metadata
I20260812 09:48:02.119297 5208 catalog_manager.cc:4841] TS 0b12969c05f24d038e309045b035f69e (127.13.99.2:39701): tablet 73421e2debc3482a88a8900d7c169268 (table test-table [id=fe09e69cdd0241cd9662f0e2e1e4dee6]) successfully deleted
I20260812 09:48:02.122143 6225 ts_tablet_manager.cc:1905] T 73421e2debc3482a88a8900d7c169268 P 5cdb8b929df74926b65c64638cbbd3f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 09:48:02.125600 6225 ts_tablet_manager.cc:1918] T 73421e2debc3482a88a8900d7c169268 P 5cdb8b929df74926b65c64638cbbd3f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 09:48:02.125738 6225 log.cc:1196] T 73421e2debc3482a88a8900d7c169268 P 5cdb8b929df74926b65c64638cbbd3f5: Deleting WAL directory at /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-2/wal/wals/73421e2debc3482a88a8900d7c169268
I20260812 09:48:02.126159 6225 ts_tablet_manager.cc:1939] T 73421e2debc3482a88a8900d7c169268 P 5cdb8b929df74926b65c64638cbbd3f5: Deleting consensus metadata
I20260812 09:48:02.127517 5209 catalog_manager.cc:4841] TS 5cdb8b929df74926b65c64638cbbd3f5 (127.13.99.3:32841): tablet 73421e2debc3482a88a8900d7c169268 (table test-table [id=fe09e69cdd0241cd9662f0e2e1e4dee6]) successfully deleted
W20260812 09:48:02.129086 5208 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.99.1:43137 (error 108)
I20260812 09:48:02.129335 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 5627
I20260812 09:48:02.135604 6171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:02.135229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.1:43137 (local address 127.0.0.1:45924)
0812 09:48:02.135414 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0812 09:48:02.135502 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.1:43137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260812 09:48:02.137603 5208 catalog_manager.cc:4584] TS a808eee3e1c14f9488cefe0d16ccc66c (127.13.99.1:43137): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 73421e2debc3482a88a8900d7c169268: Network error: Client connection negotiation failed: client connection to 127.13.99.1:43137: connect: Connection refused (error 111)
I20260812 09:48:02.144981 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 5890
I20260812 09:48:02.157922 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 5170
I20260812 09:48:02.168373 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.62:42637
--webserver_interface=127.13.99.62
--webserver_port=40349
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab
--principal=kudu/127.13.99.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:33335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.99.62:42637
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:48:02.283862 6231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:48:02.284103 6231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:48:02.284142 6231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:48:02.288384 6231 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 09:48:02.288441 6231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:48:02.288461 6231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:48:02.288538 6231 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 09:48:02.288573 6231 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 09:48:02.293907 6231 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:33335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.99.62:42637
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab
--principal=kudu/127.13.99.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.62:42637
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.99.62
--webserver_port=40349
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6231
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:48:02.294922 6231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:48:02.295961 6231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:48:02.304919 6242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:48:02.304926 6247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:48:02.304946 6243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:48:02.305454 6231 server_base.cc:1034] running on GCE node
I20260812 09:48:02.306036 6231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:48:02.307111 6231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:48:02.308314 6231 hybrid_clock.cc:648] HybridClock initialized: now 1786528082308286 us; error 50 us; skew 500 ppm
Aug 12 09:48:02 dist-test-slave-jq4t krb5kdc[4086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528082, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:48:02.311401 6231 init.cc:377] Logged in from keytab as kudu/127.13.99.62@KRBTEST.COM (short username kudu)
I20260812 09:48:02.312335 6231 webserver.cc:460] Webserver started at http://127.13.99.62:40349/ using document root <none> and password file <none>
I20260812 09:48:02.313001 6231 fs_manager.cc:359] Metadata directory not provided
I20260812 09:48:02.313102 6231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:48:02.317138 6231 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 09:48:02.319343 6262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:48:02.320648 6231 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 09:48:02.320789 6231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/master-0/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/master-0/wal
uuid: "83d811857241469db629144445c55d66"
format_stamp: "Formatted at 2026-08-12 09:47:59 on dist-test-slave-jq4t"
server_key: "fdf1de0a3059815d8e6a12d9b4a6761a"
server_key_iv: "78086bf54cfa847ca62731e158cbefed"
server_key_version: "encryptionkey@0"
I20260812 09:48:02.321336 6231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:48:02.338371 6231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:48:02.341686 6231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:48:02.341895 6231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:48:02.349011 6231 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.62:42637
I20260812 09:48:02.349294 6366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.62:42637 every 8 connection(s)
I20260812 09:48:02.350126 6231 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
I20260812 09:48:02.354416 6368 sys_catalog.cc:263] Verifying existing consensus state
I20260812 09:48:02.356027 6368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83d811857241469db629144445c55d66: Bootstrap starting.
I20260812 09:48:02.356412 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 6231
I20260812 09:48:02.356891 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.99.1:43137
--local_ip_for_outbound_sockets=127.13.99.1
--tserver_master_addrs=127.13.99.62:42637
--webserver_port=37531
--webserver_interface=127.13.99.1
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.1.keytab
--principal=kudu/127.13.99.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:33335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 09:48:02.373929 6368 log.cc:824] T 00000000000000000000000000000000 P 83d811857241469db629144445c55d66: Log is configured to *not* fsync() on all Append() calls
I20260812 09:48:02.385427 6368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83d811857241469db629144445c55d66: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 09:48:02.385811 6368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83d811857241469db629144445c55d66: Bootstrap complete.
I20260812 09:48:02.390153 6368 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83d811857241469db629144445c55d66 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83d811857241469db629144445c55d66" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 42637 } }
I20260812 09:48:02.390559 6368 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83d811857241469db629144445c55d66 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83d811857241469db629144445c55d66, State: Initialized, Role: FOLLOWER
I20260812 09:48:02.391075 6368 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83d811857241469db629144445c55d66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83d811857241469db629144445c55d66" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 42637 } }
I20260812 09:48:02.391225 6368 raft_consensus.cc:388] T 00000000000000000000000000000000 P 83d811857241469db629144445c55d66 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 09:48:02.391314 6368 raft_consensus.cc:482] T 00000000000000000000000000000000 P 83d811857241469db629144445c55d66 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 09:48:02.391433 6368 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83d811857241469db629144445c55d66 [term 2 FOLLOWER]: Advancing to term 3
I20260812 09:48:02.393355 6368 raft_consensus.cc:504] T 00000000000000000000000000000000 P 83d811857241469db629144445c55d66 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83d811857241469db629144445c55d66" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 42637 } }
I20260812 09:48:02.393690 6368 leader_election.cc:302] T 00000000000000000000000000000000 P 83d811857241469db629144445c55d66 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 83d811857241469db629144445c55d66; no voters:
I20260812 09:48:02.394052 6368 leader_election.cc:288] T 00000000000000000000000000000000 P 83d811857241469db629144445c55d66 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260812 09:48:02.394141 6376 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 83d811857241469db629144445c55d66 [term 3 FOLLOWER]: Leader election won for term 3
I20260812 09:48:02.394316 6376 raft_consensus.cc:686] T 00000000000000000000000000000000 P 83d811857241469db629144445c55d66 [term 3 LEADER]: Becoming Leader. State: Replica: 83d811857241469db629144445c55d66, State: Running, Role: LEADER
I20260812 09:48:02.394614 6376 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83d811857241469db629144445c55d66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83d811857241469db629144445c55d66" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 42637 } }
I20260812 09:48:02.395300 6368 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83d811857241469db629144445c55d66 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 09:48:02.395545 6376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83d811857241469db629144445c55d66 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83d811857241469db629144445c55d66. Latest consensus state: current_term: 3 leader_uuid: "83d811857241469db629144445c55d66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83d811857241469db629144445c55d66" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 42637 } } }
I20260812 09:48:02.395665 6376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83d811857241469db629144445c55d66 [sys.catalog]: This master's current role is: LEADER
I20260812 09:48:02.396026 6380 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 09:48:02.396155 6377 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83d811857241469db629144445c55d66 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "83d811857241469db629144445c55d66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83d811857241469db629144445c55d66" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 42637 } } }
I20260812 09:48:02.396358 6377 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83d811857241469db629144445c55d66 [sys.catalog]: This master's current role is: LEADER
I20260812 09:48:02.401436 6380 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 09:48:02.402148 6380 catalog_manager.cc:1259] Loaded cluster ID: b991061c7a9e413fa211af0ca6b9e99c
I20260812 09:48:02.402280 6380 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260812 09:48:02.403441 6380 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f4c108c39ad google::LogMessage::Fail() at ??:0
@ 0x7f4c108c688a google::LogMessage::SendToLog() at ??:0
@ 0x7f4c108c34de google::LogMessage::Flush() at ??:0
@ 0x7f4c108c4a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f4c1b202fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f4c1b203d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f4c1b200579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f4c1b244640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f4c12b153d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f4c12b15c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f4c12b173f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f4c12b06512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f4c1414a6db start_thread at ??:0
@ 0x7f4c0f3c161f clone at ??:0
W20260812 09:48:02.513736 6373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:48:02.514084 6373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:48:02.514156 6373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:48:02.520112 6373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:48:02.520191 6373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:48:02.520277 6373 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.99.1
I20260812 09:48:02.528147 6373 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:33335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.1.keytab
--principal=kudu/127.13.99.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.99.1:43137
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.99.1
--webserver_port=37531
--tserver_master_addrs=127.13.99.62:42637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6373
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.99.1
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:48:02.529569 6373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:48:02.530928 6373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:48:02.542474 6449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:48:02.542498 6447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:48:02.542666 6455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:48:02.542938 6373 server_base.cc:1034] running on GCE node
I20260812 09:48:02.543337 6373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:48:02.543979 6373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:48:02.545171 6373 hybrid_clock.cc:648] HybridClock initialized: now 1786528082545145 us; error 44 us; skew 500 ppm
Aug 12 09:48:02 dist-test-slave-jq4t krb5kdc[4086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528082, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:48:02.547881 6373 init.cc:377] Logged in from keytab as kudu/127.13.99.1@KRBTEST.COM (short username kudu)
I20260812 09:48:02.548825 6373 webserver.cc:460] Webserver started at http://127.13.99.1:37531/ using document root <none> and password file <none>
I20260812 09:48:02.549379 6373 fs_manager.cc:359] Metadata directory not provided
I20260812 09:48:02.549441 6373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:48:02.553455 6373 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 09:48:02.555785 6465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:48:02.557192 6373 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 09:48:02.557330 6373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-0/wal
uuid: "a808eee3e1c14f9488cefe0d16ccc66c"
format_stamp: "Formatted at 2026-08-12 09:48:00 on dist-test-slave-jq4t"
server_key: "cc68ccaa2e555cece62a47eb75f4b47c"
server_key_iv: "5419744aa6331d0fba4b0e14ae3946db"
server_key_version: "encryptionkey@0"
I20260812 09:48:02.557885 6373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:48:02.570698 6373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:48:02.573905 6373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:48:02.574121 6373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:48:02.574723 6373 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 09:48:02.575893 6477 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260812 09:48:02.581452 6373 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 09:48:02.581532 6373 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260812 09:48:02.581615 6373 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260812 09:48:02.583393 6373 ts_tablet_manager.cc:610] Registered 1 tablets
I20260812 09:48:02.583459 6373 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260812 09:48:02.583765 6477 tablet_bootstrap.cc:492] T 73421e2debc3482a88a8900d7c169268 P a808eee3e1c14f9488cefe0d16ccc66c: Bootstrap starting.
I20260812 09:48:02.594300 6373 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.1:43137
I20260812 09:48:02.594512 6694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.1:43137 every 8 connection(s)
I20260812 09:48:02.595355 6373 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-0/data/info.pb
I20260812 09:48:02.605774 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 6373
I20260812 09:48:02.606132 6477 log.cc:824] T 73421e2debc3482a88a8900d7c169268 P a808eee3e1c14f9488cefe0d16ccc66c: Log is configured to *not* fsync() on all Append() calls
I20260812 09:48:02.606614 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.99.2:39701
--local_ip_for_outbound_sockets=127.13.99.2
--tserver_master_addrs=127.13.99.62:42637
--webserver_port=35313
--webserver_interface=127.13.99.2
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.2.keytab
--principal=kudu/127.13.99.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:33335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 09:48:02.613714 6477 tablet_bootstrap.cc:492] T 73421e2debc3482a88a8900d7c169268 P a808eee3e1c14f9488cefe0d16ccc66c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 09:48:02.614086 6477 tablet_bootstrap.cc:492] T 73421e2debc3482a88a8900d7c169268 P a808eee3e1c14f9488cefe0d16ccc66c: Bootstrap complete.
I20260812 09:48:02.615258 6477 ts_tablet_manager.cc:1397] T 73421e2debc3482a88a8900d7c169268 P a808eee3e1c14f9488cefe0d16ccc66c: Time spent bootstrapping tablet: real 0.032s user 0.021s sys 0.007s
I20260812 09:48:02.617722 6477 raft_consensus.cc:348] T 73421e2debc3482a88a8900d7c169268 P a808eee3e1c14f9488cefe0d16ccc66c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b12969c05f24d038e309045b035f69e" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 39701 } } peers { permanent_uuid: "5cdb8b929df74926b65c64638cbbd3f5" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 32841 } } peers { permanent_uuid: "a808eee3e1c14f9488cefe0d16ccc66c" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 43137 } }
I20260812 09:48:02.618140 6477 raft_consensus.cc:729] T 73421e2debc3482a88a8900d7c169268 P a808eee3e1c14f9488cefe0d16ccc66c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a808eee3e1c14f9488cefe0d16ccc66c, State: Initialized, Role: FOLLOWER
I20260812 09:48:02.618647 6477 consensus_queue.cc:260] T 73421e2debc3482a88a8900d7c169268 P a808eee3e1c14f9488cefe0d16ccc66c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b12969c05f24d038e309045b035f69e" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 39701 } } peers { permanent_uuid: "5cdb8b929df74926b65c64638cbbd3f5" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 32841 } } peers { permanent_uuid: "a808eee3e1c14f9488cefe0d16ccc66c" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 43137 } }
I20260812 09:48:02.619514 6477 ts_tablet_manager.cc:1428] T 73421e2debc3482a88a8900d7c169268 P a808eee3e1c14f9488cefe0d16ccc66c: Time spent starting tablet: real 0.004s user 0.006s sys 0.000s
W20260812 09:48:02.660967 6701 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 09:48:02.597312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:42637 (local address 127.13.99.1:49969)
0812 09:48:02.597790 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0812 09:48:02.597843 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0812 09:48:02.598624 (+ 781us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:48:02.660155 (+ 61531us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:42637: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":151,"threads_started":1}
W20260812 09:48:02.661495 6695 heartbeater.cc:644] Failed to heartbeat to 127.13.99.62:42637 (0 consecutive failures): Network error: Failed to ping master at 127.13.99.62:42637: Client connection negotiation failed: client connection to 127.13.99.62:42637: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 09:48:02.662314 6701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:02.662072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:42637 (local address 127.13.99.1:49149)
0812 09:48:02.662202 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0812 09:48:02.662264 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:42637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260812 09:48:02.662999 6701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:02.662791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:42637 (local address 127.13.99.1:52381)
0812 09:48:02.662892 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0812 09:48:02.662946 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:42637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260812 09:48:02.663172 6695 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 09:48:02.734732 6705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:48:02.735021 6705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:48:02.735106 6705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:48:02.739087 6705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:48:02.739173 6705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:48:02.739264 6705 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.99.2
I20260812 09:48:02.744299 6705 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:33335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.2.keytab
--principal=kudu/127.13.99.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.99.2:39701
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.99.2
--webserver_port=35313
--tserver_master_addrs=127.13.99.62:42637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6705
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.99.2
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:48:02.745324 6705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:48:02.746342 6705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:48:02.756109 6716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:48:02.756148 6714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:48:02.756233 6705 server_base.cc:1034] running on GCE node
W20260812 09:48:02.756148 6713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:48:02.756880 6705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:48:02.757501 6705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:48:02.758713 6705 hybrid_clock.cc:648] HybridClock initialized: now 1786528082758698 us; error 26 us; skew 500 ppm
Aug 12 09:48:02 dist-test-slave-jq4t krb5kdc[4086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528082, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:48:02.761623 6705 init.cc:377] Logged in from keytab as kudu/127.13.99.2@KRBTEST.COM (short username kudu)
I20260812 09:48:02.762574 6705 webserver.cc:460] Webserver started at http://127.13.99.2:35313/ using document root <none> and password file <none>
I20260812 09:48:02.763151 6705 fs_manager.cc:359] Metadata directory not provided
I20260812 09:48:02.763216 6705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:48:02.767160 6705 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260812 09:48:02.769498 6733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:48:02.770785 6705 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 09:48:02.770939 6705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-1/wal
uuid: "0b12969c05f24d038e309045b035f69e"
format_stamp: "Formatted at 2026-08-12 09:48:00 on dist-test-slave-jq4t"
server_key: "4ae9a995fc90c118a5b17e256f83c06f"
server_key_iv: "6c459649e7869d8cb1627147f978999d"
server_key_version: "encryptionkey@0"
I20260812 09:48:02.771505 6705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:48:02.793968 6705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:48:02.797355 6705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:48:02.797587 6705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:48:02.798233 6705 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 09:48:02.799115 6705 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:48:02.799192 6705 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 09:48:02.799268 6705 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:48:02.799314 6705 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 09:48:02.808697 6705 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.2:39701
I20260812 09:48:02.808732 6931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.2:39701 every 8 connection(s)
I20260812 09:48:02.809754 6705 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-1/data/info.pb
I20260812 09:48:02.813886 6937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:02.811945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:42637 (local address 127.13.99.2:52307)
0812 09:48:02.812444 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0812 09:48:02.812654 (+ 210us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:42637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":135,"threads_started":1}
W20260812 09:48:02.814401 6933 heartbeater.cc:644] Failed to heartbeat to 127.13.99.62:42637 (0 consecutive failures): Network error: Failed to ping master at 127.13.99.62:42637: Client connection negotiation failed: client connection to 127.13.99.62:42637: connect: Connection refused (error 111)
I20260812 09:48:02.815003 6937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:02.814771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:42637 (local address 127.13.99.2:53169)
0812 09:48:02.814884 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0812 09:48:02.814936 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:42637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260812 09:48:02.815601 6937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:02.815387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:42637 (local address 127.13.99.2:39121)
0812 09:48:02.815496 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0812 09:48:02.815541 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:42637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260812 09:48:02.815783 6933 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260812 09:48:02.816613 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 6705
I20260812 09:48:02.817075 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.99.3:32841
--local_ip_for_outbound_sockets=127.13.99.3
--tserver_master_addrs=127.13.99.62:42637
--webserver_port=43579
--webserver_interface=127.13.99.3
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.3.keytab
--principal=kudu/127.13.99.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:33335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:48:02.928543 6940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:48:02.928791 6940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:48:02.928857 6940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:48:02.932812 6940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:48:02.932860 6940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:48:02.932973 6940 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.99.3
I20260812 09:48:02.937968 6940 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:33335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.3.keytab
--principal=kudu/127.13.99.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.99.3:32841
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.99.3
--webserver_port=43579
--tserver_master_addrs=127.13.99.62:42637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6940
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.99.3
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:48:02.938838 6940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:48:02.939852 6940 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:48:02.949186 6957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:48:02.949286 6940 server_base.cc:1034] running on GCE node
W20260812 09:48:02.949203 6950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:48:02.949467 6952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:48:02.949870 6940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:48:02.950544 6940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:48:02.951713 6940 hybrid_clock.cc:648] HybridClock initialized: now 1786528082951678 us; error 46 us; skew 500 ppm
Aug 12 09:48:02 dist-test-slave-jq4t krb5kdc[4086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528082, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:48:02.954691 6940 init.cc:377] Logged in from keytab as kudu/127.13.99.3@KRBTEST.COM (short username kudu)
I20260812 09:48:02.955586 6940 webserver.cc:460] Webserver started at http://127.13.99.3:43579/ using document root <none> and password file <none>
I20260812 09:48:02.956099 6940 fs_manager.cc:359] Metadata directory not provided
I20260812 09:48:02.956159 6940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:48:02.959856 6940 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260812 09:48:02.961956 6973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:48:02.963274 6940 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260812 09:48:02.963398 6940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-2/wal
uuid: "5cdb8b929df74926b65c64638cbbd3f5"
format_stamp: "Formatted at 2026-08-12 09:48:00 on dist-test-slave-jq4t"
server_key: "db9007ea05f29349965f91af0a28b62e"
server_key_iv: "aada7159e0a0a68d425fc6216d560d35"
server_key_version: "encryptionkey@0"
I20260812 09:48:02.963994 6940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:48:02.979328 6940 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:48:02.982223 6940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:48:02.982429 6940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:48:02.983022 6940 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 09:48:02.983867 6940 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:48:02.983946 6940 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 09:48:02.983990 6940 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:48:02.984037 6940 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 09:48:02.993105 6940 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.3:32841
I20260812 09:48:02.993181 7175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.3:32841 every 8 connection(s)
I20260812 09:48:02.994189 6940 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786527905902985-13708-0/minicluster-data/ts-2/data/info.pb
I20260812 09:48:02.997931 7185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:02.996311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:42637 (local address 127.13.99.3:58295)
0812 09:48:02.996824 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0812 09:48:02.997242 (+ 418us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:42637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":180,"threads_started":1}
W20260812 09:48:02.998466 7177 heartbeater.cc:644] Failed to heartbeat to 127.13.99.62:42637 (0 consecutive failures): Network error: Failed to ping master at 127.13.99.62:42637: Client connection negotiation failed: client connection to 127.13.99.62:42637: connect: Connection refused (error 111)
I20260812 09:48:02.999095 7185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:02.998861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:42637 (local address 127.13.99.3:33275)
0812 09:48:02.998976 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0812 09:48:02.999031 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:42637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260812 09:48:02.999750 7185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:02.999568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:42637 (local address 127.13.99.3:55775)
0812 09:48:02.999665 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0812 09:48:02.999705 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:42637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260812 09:48:02.999967 7177 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260812 09:48:03.004801 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 6940
I20260812 09:48:03.664139 7188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:03.663730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:42637 (local address 127.13.99.1:51169)
0812 09:48:03.663973 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0812 09:48:03.664027 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:42637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":75,"threads_started":1}
I20260812 09:48:03.816763 7190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:03.816339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:42637 (local address 127.13.99.2:33777)
0812 09:48:03.816617 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0812 09:48:03.816667 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:42637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":76,"threads_started":1}
I20260812 09:48:04.001011 7191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:04.000603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:42637 (local address 127.13.99.3:34205)
0812 09:48:04.000842 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0812 09:48:04.000903 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:42637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":73,"threads_started":1}
I20260812 09:48:04.323625 7194 raft_consensus.cc:482] T 73421e2debc3482a88a8900d7c169268 P a808eee3e1c14f9488cefe0d16ccc66c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 09:48:04.323854 7194 raft_consensus.cc:504] T 73421e2debc3482a88a8900d7c169268 P a808eee3e1c14f9488cefe0d16ccc66c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b12969c05f24d038e309045b035f69e" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 39701 } } peers { permanent_uuid: "5cdb8b929df74926b65c64638cbbd3f5" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 32841 } } peers { permanent_uuid: "a808eee3e1c14f9488cefe0d16ccc66c" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 43137 } }
I20260812 09:48:04.325126 7194 leader_election.cc:288] T 73421e2debc3482a88a8900d7c169268 P a808eee3e1c14f9488cefe0d16ccc66c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0b12969c05f24d038e309045b035f69e (127.13.99.2:39701), 5cdb8b929df74926b65c64638cbbd3f5 (127.13.99.3:32841)
Aug 12 09:48:04 dist-test-slave-jq4t krb5kdc[4086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528082, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.1@KRBTEST.COM for kudu/127.13.99.3@KRBTEST.COM
Aug 12 09:48:04 dist-test-slave-jq4t krb5kdc[4086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528082, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.1@KRBTEST.COM for kudu/127.13.99.2@KRBTEST.COM
I20260812 09:48:04.343112 7197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:04.325401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.3:32841 (local address 127.13.99.1:44025)
0812 09:48:04.326281 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0812 09:48:04.326296 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 09:48:04.326428 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:48:04.327683 (+ 1255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:48:04.327686 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:48:04.327972 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:48:04.328428 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:48:04.328442 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:48:04.329812 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:48:04.329816 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:48:04.330715 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:48:04.330726 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:48:04.331453 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:48:04.331457 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:48:04.331706 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:48:04.336425 (+ 4719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:48:04.336444 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:48:04.338327 (+ 1883us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:48:04.340593 (+ 2266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:48:04.340599 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:48:04.340613 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:48:04.340877 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:48:04.341309 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:48:04.341312 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:48:04.341314 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:48:04.341459 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:48:04.342135 (+ 676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:48:04.342141 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:48:04.342381 (+ 240us) client_negotiation.cc:769] Sending connection context
0812 09:48:04.342580 (+ 199us) client_negotiation.cc:240] Negotiation successful
0812 09:48:04.342910 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"mutex_wait_us":117,"spinlock_wait_cycles":2816,"thread_start_us":57,"threads_started":1}
I20260812 09:48:04.343147 7200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:04.325681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.1:44025 (local address 127.13.99.3:32841)
0812 09:48:04.326078 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0812 09:48:04.326103 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0812 09:48:04.326410 (+ 307us) server_negotiation.cc:407] Connection header received
0812 09:48:04.327076 (+ 666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:48:04.327081 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:48:04.327447 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:48:04.327734 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:48:04.328675 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:48:04.329654 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:48:04.330869 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:48:04.331325 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:48:04.331519 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:48:04.338595 (+ 7076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:48:04.338601 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:48:04.338614 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:48:04.338644 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:48:04.340414 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:48:04.341015 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:48:04.341053 (+ 38us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:48:04.341059 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:48:04.341214 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:48:04.341569 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:48:04.341574 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:48:04.341575 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:48:04.341993 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:48:04.342148 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0812 09:48:04.342705 (+ 557us) server_negotiation.cc:299] Negotiation successful
0812 09:48:04.342974 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":74,"threads_started":1}
I20260812 09:48:04.343432 7198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:04.325500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.1:52753 (local address 127.13.99.2:39701)
0812 09:48:04.325782 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0812 09:48:04.325798 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 09:48:04.325848 (+ 50us) server_negotiation.cc:407] Connection header received
0812 09:48:04.326669 (+ 821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:48:04.326676 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:48:04.327019 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:48:04.327265 (+ 246us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:48:04.328649 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:48:04.329648 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:48:04.330846 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:48:04.331323 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:48:04.331509 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:48:04.338878 (+ 7369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:48:04.338883 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:48:04.338894 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:48:04.338955 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:48:04.341085 (+ 2130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:48:04.341572 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:48:04.341578 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:48:04.341590 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:48:04.341704 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:48:04.342000 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:48:04.342004 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:48:04.342006 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:48:04.342385 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:48:04.342525 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0812 09:48:04.342895 (+ 370us) server_negotiation.cc:299] Negotiation successful
0812 09:48:04.343193 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":86,"threads_started":1}
I20260812 09:48:04.343751 7195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:04.325288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.2:39701 (local address 127.13.99.1:52753)
0812 09:48:04.325682 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0812 09:48:04.325698 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 09:48:04.325811 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:48:04.327379 (+ 1568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:48:04.327385 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:48:04.327985 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:48:04.328429 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:48:04.328445 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:48:04.329808 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:48:04.329811 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:48:04.330713 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:48:04.330723 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:48:04.331461 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:48:04.331464 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:48:04.331694 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:48:04.336473 (+ 4779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:48:04.336489 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:48:04.338675 (+ 2186us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:48:04.341234 (+ 2559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:48:04.341237 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:48:04.341239 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:48:04.341451 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:48:04.341808 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:48:04.341813 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:48:04.341815 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:48:04.341906 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:48:04.342499 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:48:04.342503 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:48:04.342604 (+ 101us) client_negotiation.cc:769] Sending connection context
0812 09:48:04.342723 (+ 119us) client_negotiation.cc:240] Negotiation successful
0812 09:48:04.343010 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"mutex_wait_us":243,"spinlock_wait_cycles":5120,"thread_start_us":70,"threads_started":1}
W20260812 09:48:04.345850 6472 leader_election.cc:341] T 73421e2debc3482a88a8900d7c169268 P a808eee3e1c14f9488cefe0d16ccc66c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5cdb8b929df74926b65c64638cbbd3f5 (127.13.99.3:32841): Not found: Tablet not found: 73421e2debc3482a88a8900d7c169268
W20260812 09:48:04.346009 6470 leader_election.cc:341] T 73421e2debc3482a88a8900d7c169268 P a808eee3e1c14f9488cefe0d16ccc66c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0b12969c05f24d038e309045b035f69e (127.13.99.2:39701): Not found: Tablet not found: 73421e2debc3482a88a8900d7c169268
I20260812 09:48:04.346129 6470 leader_election.cc:302] T 73421e2debc3482a88a8900d7c169268 P a808eee3e1c14f9488cefe0d16ccc66c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a808eee3e1c14f9488cefe0d16ccc66c; no voters: 0b12969c05f24d038e309045b035f69e, 5cdb8b929df74926b65c64638cbbd3f5
I20260812 09:48:04.346426 7194 raft_consensus.cc:2726] T 73421e2debc3482a88a8900d7c169268 P a808eee3e1c14f9488cefe0d16ccc66c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 09:48:04.665180 7195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:04.664897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:42637 (local address 127.13.99.1:48363)
0812 09:48:04.665023 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0812 09:48:04.665086 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:42637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260812 09:48:04.817983 7201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:04.817520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:42637 (local address 127.13.99.2:34953)
0812 09:48:04.817798 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0812 09:48:04.817855 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:42637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":91,"threads_started":1}
I20260812 09:48:05.002285 7203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:05.001835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:42637 (local address 127.13.99.3:50569)
0812 09:48:05.002137 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0812 09:48:05.002193 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:42637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":78,"threads_started":1}
I20260812 09:48:05.666553 7205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:05.666101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:42637 (local address 127.13.99.1:40785)
0812 09:48:05.666391 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0812 09:48:05.666446 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:42637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":121,"threads_started":1}
I20260812 09:48:05.819494 7207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:05.818870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:42637 (local address 127.13.99.2:40079)
0812 09:48:05.819273 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0812 09:48:05.819336 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:42637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":128,"threads_started":1}
I20260812 09:48:06.003589 7210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:06.003098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:42637 (local address 127.13.99.3:51315)
0812 09:48:06.003412 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0812 09:48:06.003474 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:42637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":99,"threads_started":1}
I20260812 09:48:06.190904 7211 raft_consensus.cc:482] T 73421e2debc3482a88a8900d7c169268 P a808eee3e1c14f9488cefe0d16ccc66c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 09:48:06.191053 7211 raft_consensus.cc:504] T 73421e2debc3482a88a8900d7c169268 P a808eee3e1c14f9488cefe0d16ccc66c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b12969c05f24d038e309045b035f69e" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 39701 } } peers { permanent_uuid: "5cdb8b929df74926b65c64638cbbd3f5" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 32841 } } peers { permanent_uuid: "a808eee3e1c14f9488cefe0d16ccc66c" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 43137 } }
I20260812 09:48:06.191494 7211 leader_election.cc:288] T 73421e2debc3482a88a8900d7c169268 P a808eee3e1c14f9488cefe0d16ccc66c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0b12969c05f24d038e309045b035f69e (127.13.99.2:39701), 5cdb8b929df74926b65c64638cbbd3f5 (127.13.99.3:32841)
W20260812 09:48:06.192427 6472 leader_election.cc:341] T 73421e2debc3482a88a8900d7c169268 P a808eee3e1c14f9488cefe0d16ccc66c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5cdb8b929df74926b65c64638cbbd3f5 (127.13.99.3:32841): Not found: Tablet not found: 73421e2debc3482a88a8900d7c169268
W20260812 09:48:06.192559 6470 leader_election.cc:341] T 73421e2debc3482a88a8900d7c169268 P a808eee3e1c14f9488cefe0d16ccc66c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0b12969c05f24d038e309045b035f69e (127.13.99.2:39701): Not found: Tablet not found: 73421e2debc3482a88a8900d7c169268
I20260812 09:48:06.192627 6470 leader_election.cc:302] T 73421e2debc3482a88a8900d7c169268 P a808eee3e1c14f9488cefe0d16ccc66c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a808eee3e1c14f9488cefe0d16ccc66c; no voters: 0b12969c05f24d038e309045b035f69e, 5cdb8b929df74926b65c64638cbbd3f5
I20260812 09:48:06.192755 7211 raft_consensus.cc:2726] T 73421e2debc3482a88a8900d7c169268 P a808eee3e1c14f9488cefe0d16ccc66c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 09:48:06.667771 7213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:06.667311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:42637 (local address 127.13.99.1:41039)
0812 09:48:06.667618 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0812 09:48:06.667672 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:42637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":112,"threads_started":1}
I20260812 09:48:06.820698 7217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:06.820266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:42637 (local address 127.13.99.2:50087)
0812 09:48:06.820530 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0812 09:48:06.820582 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:42637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":76,"threads_started":1}
I20260812 09:48:07.004935 7218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:07.004415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:42637 (local address 127.13.99.3:56191)
0812 09:48:07.004778 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0812 09:48:07.004834 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:42637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":166,"threads_started":1}
I20260812 09:48:07.669227 7220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:07.668763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:42637 (local address 127.13.99.1:49295)
0812 09:48:07.669070 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0812 09:48:07.669128 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:42637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":94,"threads_started":1}
I20260812 09:48:07.714257 7221 raft_consensus.cc:482] T 73421e2debc3482a88a8900d7c169268 P a808eee3e1c14f9488cefe0d16ccc66c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 09:48:07.714402 7221 raft_consensus.cc:504] T 73421e2debc3482a88a8900d7c169268 P a808eee3e1c14f9488cefe0d16ccc66c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b12969c05f24d038e309045b035f69e" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 39701 } } peers { permanent_uuid: "5cdb8b929df74926b65c64638cbbd3f5" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 32841 } } peers { permanent_uuid: "a808eee3e1c14f9488cefe0d16ccc66c" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 43137 } }
I20260812 09:48:07.714910 7221 leader_election.cc:288] T 73421e2debc3482a88a8900d7c169268 P a808eee3e1c14f9488cefe0d16ccc66c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0b12969c05f24d038e309045b035f69e (127.13.99.2:39701), 5cdb8b929df74926b65c64638cbbd3f5 (127.13.99.3:32841)
W20260812 09:48:07.715785 6470 leader_election.cc:341] T 73421e2debc3482a88a8900d7c169268 P a808eee3e1c14f9488cefe0d16ccc66c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0b12969c05f24d038e309045b035f69e (127.13.99.2:39701): Not found: Tablet not found: 73421e2debc3482a88a8900d7c169268
W20260812 09:48:07.715906 6472 leader_election.cc:341] T 73421e2debc3482a88a8900d7c169268 P a808eee3e1c14f9488cefe0d16ccc66c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5cdb8b929df74926b65c64638cbbd3f5 (127.13.99.3:32841): Not found: Tablet not found: 73421e2debc3482a88a8900d7c169268
I20260812 09:48:07.716032 6472 leader_election.cc:302] T 73421e2debc3482a88a8900d7c169268 P a808eee3e1c14f9488cefe0d16ccc66c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a808eee3e1c14f9488cefe0d16ccc66c; no voters: 0b12969c05f24d038e309045b035f69e, 5cdb8b929df74926b65c64638cbbd3f5
I20260812 09:48:07.716183 7221 raft_consensus.cc:2726] T 73421e2debc3482a88a8900d7c169268 P a808eee3e1c14f9488cefe0d16ccc66c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 09:48:07.822060 7222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:07.821595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:42637 (local address 127.13.99.2:40411)
0812 09:48:07.821879 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0812 09:48:07.821958 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:42637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":102,"threads_started":1}
I20260812 09:48:08.006117 7223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:08.005712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:42637 (local address 127.13.99.3:45023)
0812 09:48:08.005962 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0812 09:48:08.006018 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:42637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":78,"threads_started":1}
I20260812 09:48:08.670413 7225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:08.669987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:42637 (local address 127.13.99.1:44049)
0812 09:48:08.670242 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0812 09:48:08.670295 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:42637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":79,"threads_started":1}
I20260812 09:48:08.823249 7227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:08.822834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:42637 (local address 127.13.99.2:51599)
0812 09:48:08.823098 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0812 09:48:08.823154 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:42637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":93,"threads_started":1}
I20260812 09:48:09.007256 7228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:09.006828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:42637 (local address 127.13.99.3:58483)
0812 09:48:09.007096 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0812 09:48:09.007156 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:42637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":74,"threads_started":1}
I20260812 09:48:09.671651 7230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:09.671235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:42637 (local address 127.13.99.1:40607)
0812 09:48:09.671490 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0812 09:48:09.671549 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:42637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":78,"threads_started":1}
I20260812 09:48:09.824673 7231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:09.824138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:42637 (local address 127.13.99.2:59387)
0812 09:48:09.824448 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0812 09:48:09.824574 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:42637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":79,"threads_started":1}
I20260812 09:48:10.008598 7232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:10.008110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:42637 (local address 127.13.99.3:57803)
0812 09:48:10.008410 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0812 09:48:10.008503 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:42637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":78,"threads_started":1}
I20260812 09:48:10.673043 7233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:10.672649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:42637 (local address 127.13.99.1:57973)
0812 09:48:10.672884 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0812 09:48:10.672939 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:42637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":74,"threads_started":1}
I20260812 09:48:10.825933 7234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:10.825512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:42637 (local address 127.13.99.2:50653)
0812 09:48:10.825763 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0812 09:48:10.825823 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:42637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":74,"threads_started":1}
I20260812 09:48:10.944093 7235 raft_consensus.cc:482] T 73421e2debc3482a88a8900d7c169268 P a808eee3e1c14f9488cefe0d16ccc66c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 09:48:10.944272 7235 raft_consensus.cc:504] T 73421e2debc3482a88a8900d7c169268 P a808eee3e1c14f9488cefe0d16ccc66c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b12969c05f24d038e309045b035f69e" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 39701 } } peers { permanent_uuid: "5cdb8b929df74926b65c64638cbbd3f5" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 32841 } } peers { permanent_uuid: "a808eee3e1c14f9488cefe0d16ccc66c" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 43137 } }
I20260812 09:48:10.944761 7235 leader_election.cc:288] T 73421e2debc3482a88a8900d7c169268 P a808eee3e1c14f9488cefe0d16ccc66c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0b12969c05f24d038e309045b035f69e (127.13.99.2:39701), 5cdb8b929df74926b65c64638cbbd3f5 (127.13.99.3:32841)
W20260812 09:48:10.945611 6470 leader_election.cc:341] T 73421e2debc3482a88a8900d7c169268 P a808eee3e1c14f9488cefe0d16ccc66c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0b12969c05f24d038e309045b035f69e (127.13.99.2:39701): Not found: Tablet not found: 73421e2debc3482a88a8900d7c169268
W20260812 09:48:10.945721 6472 leader_election.cc:341] T 73421e2debc3482a88a8900d7c169268 P a808eee3e1c14f9488cefe0d16ccc66c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5cdb8b929df74926b65c64638cbbd3f5 (127.13.99.3:32841): Not found: Tablet not found: 73421e2debc3482a88a8900d7c169268
I20260812 09:48:10.945874 6472 leader_election.cc:302] T 73421e2debc3482a88a8900d7c169268 P a808eee3e1c14f9488cefe0d16ccc66c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a808eee3e1c14f9488cefe0d16ccc66c; no voters: 0b12969c05f24d038e309045b035f69e, 5cdb8b929df74926b65c64638cbbd3f5
I20260812 09:48:10.946048 7235 raft_consensus.cc:2726] T 73421e2debc3482a88a8900d7c169268 P a808eee3e1c14f9488cefe0d16ccc66c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 09:48:11.009786 7238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:11.009360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:42637 (local address 127.13.99.3:34941)
0812 09:48:11.009635 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0812 09:48:11.009692 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:42637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":100,"threads_started":1}
I20260812 09:48:11.674265 7240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:11.673831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:42637 (local address 127.13.99.1:58127)
0812 09:48:11.674104 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0812 09:48:11.674160 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:42637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":76,"threads_started":1}
I20260812 09:48:11.827235 7242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:11.826765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:42637 (local address 127.13.99.2:58857)
0812 09:48:11.827074 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0812 09:48:11.827135 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:42637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":85,"threads_started":1}
I20260812 09:48:12.011039 7244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:12.010577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:42637 (local address 127.13.99.3:54639)
0812 09:48:12.010861 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0812 09:48:12.010926 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:42637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":76,"threads_started":1}
I20260812 09:48:12.675711 7246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:12.675204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:42637 (local address 127.13.99.1:51259)
0812 09:48:12.675485 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0812 09:48:12.675593 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:42637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":85,"threads_started":1}
I20260812 09:48:12.828583 7248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:12.828059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:42637 (local address 127.13.99.2:42789)
0812 09:48:12.828354 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0812 09:48:12.828415 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:42637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":82,"threads_started":1}
I20260812 09:48:13.012355 7249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:13.011930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:42637 (local address 127.13.99.3:39447)
0812 09:48:13.012202 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0812 09:48:13.012258 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:42637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":79,"threads_started":1}
I20260812 09:48:13.677075 7250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:13.676654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:42637 (local address 127.13.99.1:36655)
0812 09:48:13.676910 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0812 09:48:13.676972 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:42637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":79,"threads_started":1}
I20260812 09:48:13.829835 7251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:13.829394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:42637 (local address 127.13.99.2:41563)
0812 09:48:13.829673 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0812 09:48:13.829735 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:42637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":92,"threads_started":1}
I20260812 09:48:14.013773 7253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:14.013260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:42637 (local address 127.13.99.3:33039)
0812 09:48:14.013610 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0812 09:48:14.013673 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:42637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":126,"threads_started":1}
I20260812 09:48:14.678418 7257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:14.677914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:42637 (local address 127.13.99.1:36819)
0812 09:48:14.678251 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0812 09:48:14.678315 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:42637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":100,"threads_started":1}
I20260812 09:48:14.831099 7258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:14.830650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:42637 (local address 127.13.99.2:57445)
0812 09:48:14.830941 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0812 09:48:14.831000 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:42637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":83,"threads_started":1}
I20260812 09:48:15.015095 7259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:15.014651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:42637 (local address 127.13.99.3:44785)
0812 09:48:15.014937 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0812 09:48:15.014996 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:42637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":97,"threads_started":1}
I20260812 09:48:15.679682 7261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:15.679229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:42637 (local address 127.13.99.1:38647)
0812 09:48:15.679512 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0812 09:48:15.679582 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:42637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":80,"threads_started":1}
I20260812 09:48:15.832573 7266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:15.832074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:42637 (local address 127.13.99.2:42493)
0812 09:48:15.832361 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0812 09:48:15.832412 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:42637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":91,"threads_started":1}
I20260812 09:48:16.016563 7267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:16.016002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:42637 (local address 127.13.99.3:60373)
0812 09:48:16.016329 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0812 09:48:16.016405 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:42637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":108,"threads_started":1}
I20260812 09:48:16.088074 7268 raft_consensus.cc:482] T 73421e2debc3482a88a8900d7c169268 P a808eee3e1c14f9488cefe0d16ccc66c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 09:48:16.088212 7268 raft_consensus.cc:504] T 73421e2debc3482a88a8900d7c169268 P a808eee3e1c14f9488cefe0d16ccc66c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b12969c05f24d038e309045b035f69e" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 39701 } } peers { permanent_uuid: "5cdb8b929df74926b65c64638cbbd3f5" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 32841 } } peers { permanent_uuid: "a808eee3e1c14f9488cefe0d16ccc66c" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 43137 } }
I20260812 09:48:16.088753 7268 leader_election.cc:288] T 73421e2debc3482a88a8900d7c169268 P a808eee3e1c14f9488cefe0d16ccc66c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0b12969c05f24d038e309045b035f69e (127.13.99.2:39701), 5cdb8b929df74926b65c64638cbbd3f5 (127.13.99.3:32841)
W20260812 09:48:16.089586 6470 leader_election.cc:341] T 73421e2debc3482a88a8900d7c169268 P a808eee3e1c14f9488cefe0d16ccc66c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0b12969c05f24d038e309045b035f69e (127.13.99.2:39701): Not found: Tablet not found: 73421e2debc3482a88a8900d7c169268
W20260812 09:48:16.089726 6472 leader_election.cc:341] T 73421e2debc3482a88a8900d7c169268 P a808eee3e1c14f9488cefe0d16ccc66c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5cdb8b929df74926b65c64638cbbd3f5 (127.13.99.3:32841): Not found: Tablet not found: 73421e2debc3482a88a8900d7c169268
I20260812 09:48:16.089826 6472 leader_election.cc:302] T 73421e2debc3482a88a8900d7c169268 P a808eee3e1c14f9488cefe0d16ccc66c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a808eee3e1c14f9488cefe0d16ccc66c; no voters: 0b12969c05f24d038e309045b035f69e, 5cdb8b929df74926b65c64638cbbd3f5
I20260812 09:48:16.090008 7268 raft_consensus.cc:2726] T 73421e2debc3482a88a8900d7c169268 P a808eee3e1c14f9488cefe0d16ccc66c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 09:48:16.681175 7269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:16.680640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:42637 (local address 127.13.99.1:54685)
0812 09:48:16.680986 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0812 09:48:16.681056 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:42637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":106,"threads_started":1}
I20260812 09:48:16.833702 7270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:16.833281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:42637 (local address 127.13.99.2:43205)
0812 09:48:16.833544 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0812 09:48:16.833604 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:42637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":81,"threads_started":1}
I20260812 09:48:17.017740 7271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:17.017320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:42637 (local address 127.13.99.3:42361)
0812 09:48:17.017576 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0812 09:48:17.017642 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:42637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":79,"threads_started":1}
I20260812 09:48:17.682626 7272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:17.682195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:42637 (local address 127.13.99.1:55979)
0812 09:48:17.682472 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0812 09:48:17.682529 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:42637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":76,"threads_started":1}
I20260812 09:48:17.835143 7275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:17.834667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:42637 (local address 127.13.99.2:54905)
0812 09:48:17.834958 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0812 09:48:17.835026 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:42637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":80,"threads_started":1}
I20260812 09:48:18.014622 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 6373
I20260812 09:48:18.021090 7277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:18.018847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:42637 (local address 127.13.99.3:58235)
0812 09:48:18.020927 (+ 2080us) negotiation.cc:107] Waiting for socket to connect
0812 09:48:18.020989 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:42637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1986,"thread_start_us":1815,"threads_started":1}
I20260812 09:48:18.028659 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 6705
I20260812 09:48:18.045691 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 6940
2026-08-12T09:48:18Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19843 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 09:48:18 dist-test-slave-jq4t krb5kdc[7284](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 09:48:18 dist-test-slave-jq4t krb5kdc[7284](info): set up 2 sockets
Aug 12 09:48:18 dist-test-slave-jq4t krb5kdc[7284](info): commencing operation
krb5kdc: starting...
W20260812 09:48:19.331315 13708 mini_kdc.cc:121] Time spent starting KDC: real 1.255s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/krb5kdc/test-user.keytab.
Aug 12 09:48:19 dist-test-slave-jq4t krb5kdc[7284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528099, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T09:48:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T09:48:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.99.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.99.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.99.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab.
Entry for principal kudu/127.13.99.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab.
I20260812 09:48:19.465852 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.62:37085
--webserver_interface=127.13.99.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab
--principal=kudu/127.13.99.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:39987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.99.62:37085
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:48:19.583056 7308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:48:19.583369 7308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:48:19.583442 7308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:48:19.587798 7308 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260812 09:48:19.588017 7308 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260812 09:48:19.588069 7308 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260812 09:48:19.588102 7308 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 09:48:19.588140 7308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:48:19.588181 7308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:48:19.588220 7308 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 09:48:19.588255 7308 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 09:48:19.593982 7308 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:39987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.13.99.62:37085
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab
--principal=kudu/127.13.99.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.62:37085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.99.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7308
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:48:19.595131 7308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:48:19.596264 7308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:48:19.606021 7315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:48:19.606034 7319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:48:19.606086 7308 server_base.cc:1034] running on GCE node
W20260812 09:48:19.606024 7314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:48:19.606971 7308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:48:19.608100 7308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:48:19.609272 7308 hybrid_clock.cc:648] HybridClock initialized: now 1786528099609222 us; error 65 us; skew 500 ppm
Aug 12 09:48:19 dist-test-slave-jq4t krb5kdc[7284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528099, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:48:19.612334 7308 init.cc:377] Logged in from keytab as kudu/127.13.99.62@KRBTEST.COM (short username kudu)
I20260812 09:48:19.613286 7308 webserver.cc:460] Webserver started at http://127.13.99.62:34893/ using document root <none> and password file <none>
I20260812 09:48:19.613830 7308 fs_manager.cc:359] Metadata directory not provided
I20260812 09:48:19.613900 7308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:48:19.614076 7308 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:48:19.616153 7308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/master-0/data/instance:
uuid: "6dbe64111bfb4d4a8730f47f3a1a4fb0"
format_stamp: "Formatted at 2026-08-12 09:48:19 on dist-test-slave-jq4t"
server_key: "e48ace4008e2743289311f3907fe64d7"
server_key_iv: "935ea577c777a918b942eb11a9b78c31"
server_key_version: "encryptionkey@0"
I20260812 09:48:19.616783 7308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/master-0/wal/instance:
uuid: "6dbe64111bfb4d4a8730f47f3a1a4fb0"
format_stamp: "Formatted at 2026-08-12 09:48:19 on dist-test-slave-jq4t"
server_key: "e48ace4008e2743289311f3907fe64d7"
server_key_iv: "935ea577c777a918b942eb11a9b78c31"
server_key_version: "encryptionkey@0"
I20260812 09:48:19.620656 7308 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260812 09:48:19.623255 7330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:48:19.624788 7308 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 09:48:19.624953 7308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/master-0/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/master-0/wal
uuid: "6dbe64111bfb4d4a8730f47f3a1a4fb0"
format_stamp: "Formatted at 2026-08-12 09:48:19 on dist-test-slave-jq4t"
server_key: "e48ace4008e2743289311f3907fe64d7"
server_key_iv: "935ea577c777a918b942eb11a9b78c31"
server_key_version: "encryptionkey@0"
I20260812 09:48:19.625061 7308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:48:19.643890 7308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:48:19.647346 7308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:48:19.647559 7308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:48:19.654904 7308 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.62:37085
I20260812 09:48:19.655190 7427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.62:37085 every 8 connection(s)
I20260812 09:48:19.655884 7308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
I20260812 09:48:19.658710 7436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:48:19.663064 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 7308
I20260812 09:48:19.663158 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/master-0/wal/instance
I20260812 09:48:19.663441 13708 external_mini_cluster.cc:1428] Setting key cea0e46a22c85e18a31b35132dd44efd
I20260812 09:48:19.664659 7436 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6dbe64111bfb4d4a8730f47f3a1a4fb0: Bootstrap starting.
I20260812 09:48:19.667145 7436 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6dbe64111bfb4d4a8730f47f3a1a4fb0: Neither blocks nor log segments found. Creating new log.
I20260812 09:48:19.667979 7436 log.cc:824] T 00000000000000000000000000000000 P 6dbe64111bfb4d4a8730f47f3a1a4fb0: Log is configured to *not* fsync() on all Append() calls
I20260812 09:48:19.669878 7436 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6dbe64111bfb4d4a8730f47f3a1a4fb0: No bootstrap required, opened a new log
Aug 12 09:48:19 dist-test-slave-jq4t krb5kdc[7284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528099, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
I20260812 09:48:19.674079 7436 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6dbe64111bfb4d4a8730f47f3a1a4fb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dbe64111bfb4d4a8730f47f3a1a4fb0" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 37085 } }
I20260812 09:48:19.674342 7436 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6dbe64111bfb4d4a8730f47f3a1a4fb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:48:19.674425 7436 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6dbe64111bfb4d4a8730f47f3a1a4fb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dbe64111bfb4d4a8730f47f3a1a4fb0, State: Initialized, Role: FOLLOWER
I20260812 09:48:19.674862 7436 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6dbe64111bfb4d4a8730f47f3a1a4fb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dbe64111bfb4d4a8730f47f3a1a4fb0" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 37085 } }
I20260812 09:48:19.675028 7436 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6dbe64111bfb4d4a8730f47f3a1a4fb0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 09:48:19.675134 7436 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6dbe64111bfb4d4a8730f47f3a1a4fb0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 09:48:19.675248 7436 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6dbe64111bfb4d4a8730f47f3a1a4fb0 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:48:19.676306 7436 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6dbe64111bfb4d4a8730f47f3a1a4fb0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dbe64111bfb4d4a8730f47f3a1a4fb0" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 37085 } }
I20260812 09:48:19.676731 7436 leader_election.cc:302] T 00000000000000000000000000000000 P 6dbe64111bfb4d4a8730f47f3a1a4fb0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6dbe64111bfb4d4a8730f47f3a1a4fb0; no voters:
I20260812 09:48:19.677088 7436 leader_election.cc:288] T 00000000000000000000000000000000 P 6dbe64111bfb4d4a8730f47f3a1a4fb0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 09:48:19.677292 7447 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6dbe64111bfb4d4a8730f47f3a1a4fb0 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 09:48:19.677516 7447 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6dbe64111bfb4d4a8730f47f3a1a4fb0 [term 1 LEADER]: Becoming Leader. State: Replica: 6dbe64111bfb4d4a8730f47f3a1a4fb0, State: Running, Role: LEADER
I20260812 09:48:19.677834 7447 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6dbe64111bfb4d4a8730f47f3a1a4fb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dbe64111bfb4d4a8730f47f3a1a4fb0" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 37085 } }
I20260812 09:48:19.678153 7443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:19.664922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39592 (local address 127.13.99.62:37085)
0812 09:48:19.665328 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0812 09:48:19.665344 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 09:48:19.665379 (+ 35us) server_negotiation.cc:407] Connection header received
0812 09:48:19.665987 (+ 608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:48:19.666003 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:48:19.666322 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:48:19.666584 (+ 262us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:48:19.667572 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:48:19.668695 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:48:19.669593 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:48:19.670116 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:48:19.670294 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:48:19.673012 (+ 2718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:48:19.673018 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:48:19.673034 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:48:19.673061 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:48:19.674970 (+ 1909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:48:19.675527 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:48:19.675532 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:48:19.675536 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:48:19.675644 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:48:19.675956 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:48:19.675958 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:48:19.675959 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:48:19.676355 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:48:19.676657 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0812 09:48:19.676894 (+ 237us) server_negotiation.cc:299] Negotiation successful
0812 09:48:19.677231 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":135,"threads_started":1}
I20260812 09:48:19.678418 7436 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6dbe64111bfb4d4a8730f47f3a1a4fb0 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 09:48:19.679052 7447 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dbe64111bfb4d4a8730f47f3a1a4fb0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6dbe64111bfb4d4a8730f47f3a1a4fb0. Latest consensus state: current_term: 1 leader_uuid: "6dbe64111bfb4d4a8730f47f3a1a4fb0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dbe64111bfb4d4a8730f47f3a1a4fb0" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 37085 } } }
I20260812 09:48:19.679177 7447 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dbe64111bfb4d4a8730f47f3a1a4fb0 [sys.catalog]: This master's current role is: LEADER
I20260812 09:48:19.680404 7453 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 09:48:19.681596 7448 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dbe64111bfb4d4a8730f47f3a1a4fb0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6dbe64111bfb4d4a8730f47f3a1a4fb0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dbe64111bfb4d4a8730f47f3a1a4fb0" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 37085 } } }
I20260812 09:48:19.681712 7448 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dbe64111bfb4d4a8730f47f3a1a4fb0 [sys.catalog]: This master's current role is: LEADER
I20260812 09:48:19.683516 7453 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 09:48:19.689330 7453 catalog_manager.cc:1347] Generated new cluster ID: 68a068ddd80a4047bf4e04d78077accc
I20260812 09:48:19.689410 7453 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 09:48:19.705174 7453 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 09:48:19.706202 7453 catalog_manager.cc:1504] Loading token signing keys...
I20260812 09:48:19.715974 7453 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6dbe64111bfb4d4a8730f47f3a1a4fb0: Generated new TSK 0
I20260812 09:48:19.716775 7453 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.99.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.99.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.99.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.1.keytab.
Entry for principal kudu/127.13.99.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.1.keytab.
I20260812 09:48:19.756109 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.99.1:0
--local_ip_for_outbound_sockets=127.13.99.1
--webserver_interface=127.13.99.1
--webserver_port=0
--tserver_master_addrs=127.13.99.62:37085
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.1.keytab
--principal=kudu/127.13.99.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:39987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:48:19.891069 7478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:48:19.891324 7478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:48:19.891367 7478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:48:19.895414 7478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:48:19.895470 7478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:48:19.895526 7478 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.99.1
I20260812 09:48:19.900588 7478 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:39987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.1.keytab
--principal=kudu/127.13.99.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.99.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.99.1
--webserver_port=0
--tserver_master_addrs=127.13.99.62:37085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.99.1
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:48:19.901539 7478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:48:19.902508 7478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:48:19.912436 7486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:48:19.912537 7488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:48:19.912436 7485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:48:19.912743 7478 server_base.cc:1034] running on GCE node
I20260812 09:48:19.913200 7478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:48:19.913877 7478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:48:19.915071 7478 hybrid_clock.cc:648] HybridClock initialized: now 1786528099915057 us; error 26 us; skew 500 ppm
Aug 12 09:48:19 dist-test-slave-jq4t krb5kdc[7284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528099, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:48:19.917860 7478 init.cc:377] Logged in from keytab as kudu/127.13.99.1@KRBTEST.COM (short username kudu)
I20260812 09:48:19.918711 7478 webserver.cc:460] Webserver started at http://127.13.99.1:36855/ using document root <none> and password file <none>
I20260812 09:48:19.919283 7478 fs_manager.cc:359] Metadata directory not provided
I20260812 09:48:19.919379 7478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:48:19.919622 7478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:48:19.921694 7478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-0/data/instance:
uuid: "f9e2dce914b843b2bfc0de5f215a45ff"
format_stamp: "Formatted at 2026-08-12 09:48:19 on dist-test-slave-jq4t"
server_key: "493d29e9ebcdf3ef9dadeb75241b9724"
server_key_iv: "322088c3e4178e5fd7efe60477020e3b"
server_key_version: "encryptionkey@0"
I20260812 09:48:19.922259 7478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-0/wal/instance:
uuid: "f9e2dce914b843b2bfc0de5f215a45ff"
format_stamp: "Formatted at 2026-08-12 09:48:19 on dist-test-slave-jq4t"
server_key: "493d29e9ebcdf3ef9dadeb75241b9724"
server_key_iv: "322088c3e4178e5fd7efe60477020e3b"
server_key_version: "encryptionkey@0"
I20260812 09:48:19.925944 7478 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 09:48:19.928264 7499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:48:19.929535 7478 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 09:48:19.929728 7478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-0/wal
uuid: "f9e2dce914b843b2bfc0de5f215a45ff"
format_stamp: "Formatted at 2026-08-12 09:48:19 on dist-test-slave-jq4t"
server_key: "493d29e9ebcdf3ef9dadeb75241b9724"
server_key_iv: "322088c3e4178e5fd7efe60477020e3b"
server_key_version: "encryptionkey@0"
I20260812 09:48:19.929857 7478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:48:19.945811 7478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:48:19.949098 7478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:48:19.949308 7478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:48:19.949925 7478 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 09:48:19.950793 7478 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:48:19.950883 7478 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 09:48:19.950958 7478 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:48:19.951001 7478 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 09:48:19.960464 7478 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.1:34429
I20260812 09:48:19.960834 7693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.1:34429 every 8 connection(s)
I20260812 09:48:19.961444 7478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-0/data/info.pb
I20260812 09:48:19.964329 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 7478
I20260812 09:48:19.964457 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-0/wal/instance
I20260812 09:48:19.964790 13708 external_mini_cluster.cc:1428] Setting key 631703c3c1e7d9c5b787c15f0e31bd0e
Aug 12 09:48:19 dist-test-slave-jq4t krb5kdc[7284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528099, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.1@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
I20260812 09:48:19.976445 7443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:19.963295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.1:58597 (local address 127.13.99.62:37085)
0812 09:48:19.963471 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0812 09:48:19.963476 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:48:19.964303 (+ 827us) server_negotiation.cc:407] Connection header received
0812 09:48:19.965447 (+ 1144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:48:19.965451 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:48:19.965507 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:48:19.965620 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:48:19.967072 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:48:19.967924 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:48:19.969214 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:48:19.969458 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:48:19.969580 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:48:19.972307 (+ 2727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:48:19.972311 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:48:19.972314 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:48:19.972345 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:48:19.974107 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:48:19.974655 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:48:19.974660 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:48:19.974663 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:48:19.974763 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:48:19.975128 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:48:19.975133 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:48:19.975136 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:48:19.975396 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:48:19.975531 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0812 09:48:19.976064 (+ 533us) server_negotiation.cc:299] Negotiation successful
0812 09:48:19.976218 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
WARNING: no policy specified for kudu/127.13.99.2@KRBTEST.COM; defaulting to no policy
I20260812 09:48:19.977177 7702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:19.963684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:37085 (local address 127.13.99.1:58597)
0812 09:48:19.964147 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0812 09:48:19.964189 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 09:48:19.965210 (+ 1021us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:48:19.965822 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:48:19.965831 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:48:19.966359 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:48:19.966889 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:48:19.966903 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:48:19.968061 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:48:19.968064 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:48:19.969071 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:48:19.969081 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:48:19.969620 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:48:19.969624 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:48:19.969798 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:48:19.970423 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:48:19.970450 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:48:19.972129 (+ 1679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:48:19.974244 (+ 2115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:48:19.974249 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:48:19.974260 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:48:19.974529 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:48:19.974879 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:48:19.974882 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:48:19.974883 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:48:19.975017 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:48:19.975514 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:48:19.975519 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:48:19.975804 (+ 285us) client_negotiation.cc:769] Sending connection context
0812 09:48:19.975965 (+ 161us) client_negotiation.cc:240] Negotiation successful
0812 09:48:19.976215 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":115,"threads_started":1}
I20260812 09:48:19.978581 7697 heartbeater.cc:347] Connected to a master server at 127.13.99.62:37085
I20260812 09:48:19.978834 7697 heartbeater.cc:464] Registering TS with master...
I20260812 09:48:19.979393 7697 heartbeater.cc:511] Master 127.13.99.62:37085 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.99.2@KRBTEST.COM" created.
I20260812 09:48:19.981297 7349 ts_manager.cc:194] Registered new tserver with Master: f9e2dce914b843b2bfc0de5f215a45ff (127.13.99.1:34429)
I20260812 09:48:19.982574 7349 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.99.1@KRBTEST.COM'} at 127.13.99.1:58597
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.99.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.2.keytab.
Entry for principal kudu/127.13.99.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.2.keytab.
I20260812 09:48:19.997601 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.99.2:0
--local_ip_for_outbound_sockets=127.13.99.2
--webserver_interface=127.13.99.2
--webserver_port=0
--tserver_master_addrs=127.13.99.62:37085
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.2.keytab
--principal=kudu/127.13.99.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:39987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:48:20.111881 7706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:48:20.112120 7706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:48:20.112161 7706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:48:20.116180 7706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:48:20.116236 7706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:48:20.116291 7706 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.99.2
I20260812 09:48:20.121357 7706 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:39987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.2.keytab
--principal=kudu/127.13.99.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.99.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.99.2
--webserver_port=0
--tserver_master_addrs=127.13.99.62:37085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7706
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.99.2
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:48:20.122295 7706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:48:20.123252 7706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 09:48:20.133316 7706 server_base.cc:1034] running on GCE node
W20260812 09:48:20.133279 7718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:48:20.133266 7722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:48:20.133279 7719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:48:20.133999 7706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:48:20.134570 7706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:48:20.135749 7706 hybrid_clock.cc:648] HybridClock initialized: now 1786528100135728 us; error 33 us; skew 500 ppm
Aug 12 09:48:20 dist-test-slave-jq4t krb5kdc[7284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528100, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:48:20.138643 7706 init.cc:377] Logged in from keytab as kudu/127.13.99.2@KRBTEST.COM (short username kudu)
I20260812 09:48:20.139559 7706 webserver.cc:460] Webserver started at http://127.13.99.2:37537/ using document root <none> and password file <none>
I20260812 09:48:20.140148 7706 fs_manager.cc:359] Metadata directory not provided
I20260812 09:48:20.140249 7706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:48:20.140554 7706 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:48:20.142643 7706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-1/data/instance:
uuid: "90b746efe2d54317b04ae5164cef1a3b"
format_stamp: "Formatted at 2026-08-12 09:48:20 on dist-test-slave-jq4t"
server_key: "2304f1d32d8d1d9ae92fb41ef8950a57"
server_key_iv: "98ab55e79a37ce873231623ae8b1522c"
server_key_version: "encryptionkey@0"
I20260812 09:48:20.143240 7706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-1/wal/instance:
uuid: "90b746efe2d54317b04ae5164cef1a3b"
format_stamp: "Formatted at 2026-08-12 09:48:20 on dist-test-slave-jq4t"
server_key: "2304f1d32d8d1d9ae92fb41ef8950a57"
server_key_iv: "98ab55e79a37ce873231623ae8b1522c"
server_key_version: "encryptionkey@0"
I20260812 09:48:20.146956 7706 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260812 09:48:20.149542 7733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:48:20.150758 7706 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 09:48:20.150900 7706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-1/wal
uuid: "90b746efe2d54317b04ae5164cef1a3b"
format_stamp: "Formatted at 2026-08-12 09:48:20 on dist-test-slave-jq4t"
server_key: "2304f1d32d8d1d9ae92fb41ef8950a57"
server_key_iv: "98ab55e79a37ce873231623ae8b1522c"
server_key_version: "encryptionkey@0"
I20260812 09:48:20.151013 7706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:48:20.189715 7706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:48:20.192837 7706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:48:20.193056 7706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:48:20.193672 7706 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 09:48:20.194532 7706 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:48:20.194610 7706 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 09:48:20.194679 7706 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:48:20.194730 7706 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 09:48:20.204039 7706 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.2:43737
I20260812 09:48:20.204066 7953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.2:43737 every 8 connection(s)
I20260812 09:48:20.205035 7706 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-1/data/info.pb
Aug 12 09:48:20 dist-test-slave-jq4t krb5kdc[7284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528100, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.2@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
I20260812 09:48:20.215235 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 7706
I20260812 09:48:20.215382 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-1/wal/instance
I20260812 09:48:20.215674 13708 external_mini_cluster.cc:1428] Setting key 092edbf907a737b0c3059e34d2bf207d
I20260812 09:48:20.219653 7443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:20.206896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.2:42637 (local address 127.13.99.62:37085)
0812 09:48:20.207046 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0812 09:48:20.207050 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:48:20.207865 (+ 815us) server_negotiation.cc:407] Connection header received
0812 09:48:20.209334 (+ 1469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:48:20.209338 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:48:20.209396 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:48:20.209514 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:48:20.210702 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:48:20.211276 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:48:20.212449 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:48:20.212728 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:48:20.212843 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:48:20.215360 (+ 2517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:48:20.215364 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:48:20.215368 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:48:20.215405 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:48:20.217299 (+ 1894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:48:20.217936 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:48:20.217939 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:48:20.217940 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:48:20.218017 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:48:20.218447 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:48:20.218449 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:48:20.218451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:48:20.218637 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:48:20.218714 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0812 09:48:20.219392 (+ 678us) server_negotiation.cc:299] Negotiation successful
0812 09:48:20.219493 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260812 09:48:20.220897 7958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:20.207260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:37085 (local address 127.13.99.2:42637)
0812 09:48:20.207717 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0812 09:48:20.207759 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 09:48:20.209118 (+ 1359us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:48:20.209694 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:48:20.209701 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:48:20.210072 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:48:20.210552 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:48:20.210566 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:48:20.211416 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:48:20.211419 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:48:20.212313 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:48:20.212323 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:48:20.212897 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:48:20.212901 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:48:20.213075 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:48:20.213640 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:48:20.213656 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:48:20.215190 (+ 1534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:48:20.217429 (+ 2239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:48:20.217435 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:48:20.217450 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:48:20.217824 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:48:20.218121 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:48:20.218126 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:48:20.218146 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:48:20.218345 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:48:20.218748 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:48:20.218757 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:48:20.219121 (+ 364us) client_negotiation.cc:769] Sending connection context
0812 09:48:20.219349 (+ 228us) client_negotiation.cc:240] Negotiation successful
0812 09:48:20.220034 (+ 685us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":113,"threads_started":1}
I20260812 09:48:20.222152 7954 heartbeater.cc:347] Connected to a master server at 127.13.99.62:37085
I20260812 09:48:20.222419 7954 heartbeater.cc:464] Registering TS with master...
I20260812 09:48:20.223091 7954 heartbeater.cc:511] Master 127.13.99.62:37085 requested a full tablet report, sending...
I20260812 09:48:20.224299 7349 ts_manager.cc:194] Registered new tserver with Master: 90b746efe2d54317b04ae5164cef1a3b (127.13.99.2:43737)
I20260812 09:48:20.224942 7349 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.99.2@KRBTEST.COM'} at 127.13.99.2:42637
WARNING: no policy specified for kudu/127.13.99.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.99.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.99.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.3.keytab.
Entry for principal kudu/127.13.99.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.3.keytab.
I20260812 09:48:20.247426 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.99.3:0
--local_ip_for_outbound_sockets=127.13.99.3
--webserver_interface=127.13.99.3
--webserver_port=0
--tserver_master_addrs=127.13.99.62:37085
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.3.keytab
--principal=kudu/127.13.99.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:39987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:48:20.362512 7961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:48:20.362803 7961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:48:20.362887 7961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:48:20.366923 7961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:48:20.367013 7961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:48:20.367105 7961 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.99.3
I20260812 09:48:20.372268 7961 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:39987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.3.keytab
--principal=kudu/127.13.99.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.99.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.99.3
--webserver_port=0
--tserver_master_addrs=127.13.99.62:37085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7961
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.99.3
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:48:20.373417 7961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:48:20.374483 7961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:48:20.384742 7971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:48:20.384738 7970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:48:20.384783 7974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:48:20.384814 7961 server_base.cc:1034] running on GCE node
I20260812 09:48:20.385469 7961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:48:20.386148 7961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:48:20.387358 7961 hybrid_clock.cc:648] HybridClock initialized: now 1786528100387343 us; error 37 us; skew 500 ppm
Aug 12 09:48:20 dist-test-slave-jq4t krb5kdc[7284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528100, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:48:20.390352 7961 init.cc:377] Logged in from keytab as kudu/127.13.99.3@KRBTEST.COM (short username kudu)
I20260812 09:48:20.391285 7961 webserver.cc:460] Webserver started at http://127.13.99.3:34537/ using document root <none> and password file <none>
I20260812 09:48:20.391887 7961 fs_manager.cc:359] Metadata directory not provided
I20260812 09:48:20.391987 7961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:48:20.392228 7961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:48:20.394389 7961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-2/data/instance:
uuid: "6440e49253344afebefac34f6e39e878"
format_stamp: "Formatted at 2026-08-12 09:48:20 on dist-test-slave-jq4t"
server_key: "523a68247cb62087c9374fbdf31e43b3"
server_key_iv: "c08b041a9e4a37ba82a2e3f18c15f5fb"
server_key_version: "encryptionkey@0"
I20260812 09:48:20.394981 7961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-2/wal/instance:
uuid: "6440e49253344afebefac34f6e39e878"
format_stamp: "Formatted at 2026-08-12 09:48:20 on dist-test-slave-jq4t"
server_key: "523a68247cb62087c9374fbdf31e43b3"
server_key_iv: "c08b041a9e4a37ba82a2e3f18c15f5fb"
server_key_version: "encryptionkey@0"
I20260812 09:48:20.398806 7961 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260812 09:48:20.401298 7986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:48:20.402544 7961 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 09:48:20.402694 7961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-2/wal
uuid: "6440e49253344afebefac34f6e39e878"
format_stamp: "Formatted at 2026-08-12 09:48:20 on dist-test-slave-jq4t"
server_key: "523a68247cb62087c9374fbdf31e43b3"
server_key_iv: "c08b041a9e4a37ba82a2e3f18c15f5fb"
server_key_version: "encryptionkey@0"
I20260812 09:48:20.402809 7961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:48:20.416671 7961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:48:20.419967 7961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:48:20.420181 7961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:48:20.420830 7961 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 09:48:20.421675 7961 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:48:20.421751 7961 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 09:48:20.421818 7961 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:48:20.421877 7961 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 09:48:20.430866 7961 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.3:40441
I20260812 09:48:20.430919 8194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.3:40441 every 8 connection(s)
I20260812 09:48:20.431783 7961 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-2/data/info.pb
I20260812 09:48:20.434374 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 7961
I20260812 09:48:20.434530 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-2/wal/instance
I20260812 09:48:20.434855 13708 external_mini_cluster.cc:1428] Setting key 7810420e569c0aade31d6597d9346999
Aug 12 09:48:20 dist-test-slave-jq4t krb5kdc[7284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528100, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.3@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
I20260812 09:48:20.446267 7443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:20.433849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.3:44143 (local address 127.13.99.62:37085)
0812 09:48:20.433991 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0812 09:48:20.433996 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:48:20.434924 (+ 928us) server_negotiation.cc:407] Connection header received
0812 09:48:20.436357 (+ 1433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:48:20.436363 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:48:20.436422 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:48:20.436565 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:48:20.437881 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:48:20.438443 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:48:20.439693 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:48:20.439922 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:48:20.440018 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:48:20.442766 (+ 2748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:48:20.442769 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:48:20.442771 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:48:20.442798 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:48:20.444121 (+ 1323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:48:20.444797 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:48:20.444800 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:48:20.444802 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:48:20.444890 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:48:20.445256 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:48:20.445259 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:48:20.445260 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:48:20.445458 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:48:20.445546 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0812 09:48:20.446041 (+ 495us) server_negotiation.cc:299] Negotiation successful
0812 09:48:20.446143 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260812 09:48:20.447103 8199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:20.434216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:37085 (local address 127.13.99.3:44143)
0812 09:48:20.434752 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0812 09:48:20.434805 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0812 09:48:20.436163 (+ 1358us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:48:20.436728 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:48:20.436735 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:48:20.437111 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:48:20.437707 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:48:20.437723 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:48:20.438601 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:48:20.438604 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:48:20.439522 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:48:20.439532 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:48:20.440066 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:48:20.440068 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:48:20.440208 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:48:20.440917 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:48:20.440934 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:48:20.442596 (+ 1662us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:48:20.444249 (+ 1653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:48:20.444254 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:48:20.444267 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:48:20.444644 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:48:20.444990 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:48:20.444993 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:48:20.444994 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:48:20.445143 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:48:20.445564 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:48:20.445569 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:48:20.445812 (+ 243us) client_negotiation.cc:769] Sending connection context
0812 09:48:20.445982 (+ 170us) client_negotiation.cc:240] Negotiation successful
0812 09:48:20.446249 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":154,"threads_started":1}
I20260812 09:48:20.448338 8195 heartbeater.cc:347] Connected to a master server at 127.13.99.62:37085
I20260812 09:48:20.448632 8195 heartbeater.cc:464] Registering TS with master...
I20260812 09:48:20.449206 8195 heartbeater.cc:511] Master 127.13.99.62:37085 requested a full tablet report, sending...
I20260812 09:48:20.450285 7349 ts_manager.cc:194] Registered new tserver with Master: 6440e49253344afebefac34f6e39e878 (127.13.99.3:40441)
I20260812 09:48:20.450870 7349 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.99.3@KRBTEST.COM'} at 127.13.99.3:44143
I20260812 09:48:20.461228 13708 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 09:48:20.470841 7443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:20.463327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39594 (local address 127.13.99.62:37085)
0812 09:48:20.463510 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0812 09:48:20.463513 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 09:48:20.463578 (+ 65us) server_negotiation.cc:407] Connection header received
0812 09:48:20.463705 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:48:20.463707 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:48:20.463747 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:48:20.463813 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:48:20.464668 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:48:20.465178 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:48:20.466026 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:48:20.466231 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:48:20.466308 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:48:20.467744 (+ 1436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:48:20.467747 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:48:20.467749 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:48:20.467767 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:48:20.469265 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:48:20.469676 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:48:20.469678 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:48:20.469680 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:48:20.469755 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:48:20.469993 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:48:20.469995 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:48:20.469996 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:48:20.470199 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:48:20.470304 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0812 09:48:20.470516 (+ 212us) server_negotiation.cc:299] Negotiation successful
0812 09:48:20.470609 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260812 09:48:20.473716 7349 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39594:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 09:48:20.476819 7349 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 09:48:20.492102 8222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:20.486617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.2:43737 (local address 127.0.0.1:37254)
0812 09:48:20.486858 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0812 09:48:20.486871 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 09:48:20.486958 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:48:20.488266 (+ 1308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:48:20.488269 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:48:20.488288 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:48:20.488366 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:48:20.488373 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:48:20.489678 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:48:20.489681 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:48:20.490617 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:48:20.490628 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:48:20.491557 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:48:20.491561 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:48:20.491667 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:48:20.491704 (+ 37us) client_negotiation.cc:769] Sending connection context
0812 09:48:20.491816 (+ 112us) client_negotiation.cc:240] Negotiation successful
0812 09:48:20.491932 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":2,"thread_start_us":106,"threads_started":1}
I20260812 09:48:20.492153 8217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:20.485978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.3:40441 (local address 127.0.0.1:53146)
0812 09:48:20.486567 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0812 09:48:20.486580 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 09:48:20.486647 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:48:20.487394 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:48:20.487401 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:48:20.487423 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:48:20.487578 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:48:20.487587 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:48:20.489123 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:48:20.489127 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:48:20.490747 (+ 1620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:48:20.490759 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:48:20.491823 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:48:20.491826 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:48:20.491907 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:48:20.491928 (+ 21us) client_negotiation.cc:769] Sending connection context
0812 09:48:20.491986 (+ 58us) client_negotiation.cc:240] Negotiation successful
0812 09:48:20.492047 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"mutex_wait_us":128,"thread_start_us":149,"threads_started":1}
I20260812 09:48:20.492471 8219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:20.486155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53146 (local address 127.13.99.3:40441)
0812 09:48:20.486927 (+ 772us) server_negotiation.cc:206] Beginning negotiation
0812 09:48:20.486941 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 09:48:20.486965 (+ 24us) server_negotiation.cc:407] Connection header received
0812 09:48:20.487061 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:48:20.487066 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:48:20.487233 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:48:20.487402 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:48:20.487734 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:48:20.488963 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:48:20.490914 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:48:20.491710 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:48:20.491831 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:48:20.491971 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0812 09:48:20.492024 (+ 53us) server_negotiation.cc:299] Negotiation successful
0812 09:48:20.492119 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":660,"thread_start_us":102,"threads_started":1}
I20260812 09:48:20.492636 8226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:20.487234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37254 (local address 127.13.99.2:43737)
0812 09:48:20.487731 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0812 09:48:20.487744 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 09:48:20.487765 (+ 21us) server_negotiation.cc:407] Connection header received
0812 09:48:20.487840 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:48:20.487846 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:48:20.488032 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:48:20.488179 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:48:20.488523 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:48:20.489536 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:48:20.490758 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:48:20.491414 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:48:20.491559 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:48:20.491708 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0812 09:48:20.492303 (+ 595us) server_negotiation.cc:299] Negotiation successful
0812 09:48:20.492404 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":111,"threads_started":1}
I20260812 09:48:20.493018 8223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:20.486842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42188 (local address 127.13.99.1:34429)
0812 09:48:20.487156 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0812 09:48:20.487163 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 09:48:20.487188 (+ 25us) server_negotiation.cc:407] Connection header received
0812 09:48:20.487265 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:48:20.487270 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:48:20.487464 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:48:20.487943 (+ 479us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:48:20.488091 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:48:20.488976 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:48:20.490914 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:48:20.491710 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:48:20.492616 (+ 906us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:48:20.492771 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0812 09:48:20.492839 (+ 68us) server_negotiation.cc:299] Negotiation successful
0812 09:48:20.492911 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":140,"threads_started":1}
I20260812 09:48:20.493505 8218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:20.485856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.1:34429 (local address 127.0.0.1:42188)
0812 09:48:20.486406 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0812 09:48:20.486455 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0812 09:48:20.486602 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:48:20.487650 (+ 1048us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:48:20.487656 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:48:20.487680 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:48:20.487786 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:48:20.487795 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:48:20.489097 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:48:20.489102 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:48:20.490747 (+ 1645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:48:20.490759 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:48:20.491855 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:48:20.491858 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:48:20.491960 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:48:20.491986 (+ 26us) client_negotiation.cc:769] Sending connection context
0812 09:48:20.492054 (+ 68us) client_negotiation.cc:240] Negotiation successful
0812 09:48:20.492120 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"mutex_wait_us":186,"thread_start_us":108,"threads_started":1}
I20260812 09:48:20.495380 8066 tablet_service.cc:1467] Processing CreateTablet for tablet 136aadace1724175a669350f88ff6dbd (DEFAULT_TABLE table=test-table [id=c569756a8c664219a2b8f04917450b68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 09:48:20.495744 7577 tablet_service.cc:1467] Processing CreateTablet for tablet 136aadace1724175a669350f88ff6dbd (DEFAULT_TABLE table=test-table [id=c569756a8c664219a2b8f04917450b68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 09:48:20.496428 8066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 136aadace1724175a669350f88ff6dbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:48:20.496724 7577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 136aadace1724175a669350f88ff6dbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:48:20.501854 7833 tablet_service.cc:1467] Processing CreateTablet for tablet 136aadace1724175a669350f88ff6dbd (DEFAULT_TABLE table=test-table [id=c569756a8c664219a2b8f04917450b68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 09:48:20.502471 8228 tablet_bootstrap.cc:492] T 136aadace1724175a669350f88ff6dbd P 6440e49253344afebefac34f6e39e878: Bootstrap starting.
I20260812 09:48:20.502905 7833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 136aadace1724175a669350f88ff6dbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:48:20.504242 8229 tablet_bootstrap.cc:492] T 136aadace1724175a669350f88ff6dbd P f9e2dce914b843b2bfc0de5f215a45ff: Bootstrap starting.
I20260812 09:48:20.504560 8228 tablet_bootstrap.cc:654] T 136aadace1724175a669350f88ff6dbd P 6440e49253344afebefac34f6e39e878: Neither blocks nor log segments found. Creating new log.
I20260812 09:48:20.505841 8228 log.cc:824] T 136aadace1724175a669350f88ff6dbd P 6440e49253344afebefac34f6e39e878: Log is configured to *not* fsync() on all Append() calls
I20260812 09:48:20.506429 8229 tablet_bootstrap.cc:654] T 136aadace1724175a669350f88ff6dbd P f9e2dce914b843b2bfc0de5f215a45ff: Neither blocks nor log segments found. Creating new log.
I20260812 09:48:20.507339 8229 log.cc:824] T 136aadace1724175a669350f88ff6dbd P f9e2dce914b843b2bfc0de5f215a45ff: Log is configured to *not* fsync() on all Append() calls
I20260812 09:48:20.507869 8228 tablet_bootstrap.cc:492] T 136aadace1724175a669350f88ff6dbd P 6440e49253344afebefac34f6e39e878: No bootstrap required, opened a new log
I20260812 09:48:20.508075 8228 ts_tablet_manager.cc:1397] T 136aadace1724175a669350f88ff6dbd P 6440e49253344afebefac34f6e39e878: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260812 09:48:20.509634 8229 tablet_bootstrap.cc:492] T 136aadace1724175a669350f88ff6dbd P f9e2dce914b843b2bfc0de5f215a45ff: No bootstrap required, opened a new log
I20260812 09:48:20.509876 8229 ts_tablet_manager.cc:1397] T 136aadace1724175a669350f88ff6dbd P f9e2dce914b843b2bfc0de5f215a45ff: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260812 09:48:20.509981 8234 tablet_bootstrap.cc:492] T 136aadace1724175a669350f88ff6dbd P 90b746efe2d54317b04ae5164cef1a3b: Bootstrap starting.
I20260812 09:48:20.511504 8228 raft_consensus.cc:348] T 136aadace1724175a669350f88ff6dbd P 6440e49253344afebefac34f6e39e878 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9e2dce914b843b2bfc0de5f215a45ff" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 34429 } } peers { permanent_uuid: "6440e49253344afebefac34f6e39e878" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 40441 } } peers { permanent_uuid: "90b746efe2d54317b04ae5164cef1a3b" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 43737 } }
I20260812 09:48:20.511794 8228 raft_consensus.cc:374] T 136aadace1724175a669350f88ff6dbd P 6440e49253344afebefac34f6e39e878 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:48:20.511853 8228 raft_consensus.cc:729] T 136aadace1724175a669350f88ff6dbd P 6440e49253344afebefac34f6e39e878 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6440e49253344afebefac34f6e39e878, State: Initialized, Role: FOLLOWER
I20260812 09:48:20.512300 8228 consensus_queue.cc:260] T 136aadace1724175a669350f88ff6dbd P 6440e49253344afebefac34f6e39e878 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9e2dce914b843b2bfc0de5f215a45ff" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 34429 } } peers { permanent_uuid: "6440e49253344afebefac34f6e39e878" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 40441 } } peers { permanent_uuid: "90b746efe2d54317b04ae5164cef1a3b" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 43737 } }
I20260812 09:48:20.513033 8195 heartbeater.cc:503] Master 127.13.99.62:37085 was elected leader, sending a full tablet report...
I20260812 09:48:20.513175 8234 tablet_bootstrap.cc:654] T 136aadace1724175a669350f88ff6dbd P 90b746efe2d54317b04ae5164cef1a3b: Neither blocks nor log segments found. Creating new log.
I20260812 09:48:20.513458 8228 ts_tablet_manager.cc:1428] T 136aadace1724175a669350f88ff6dbd P 6440e49253344afebefac34f6e39e878: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260812 09:48:20.514014 8234 log.cc:824] T 136aadace1724175a669350f88ff6dbd P 90b746efe2d54317b04ae5164cef1a3b: Log is configured to *not* fsync() on all Append() calls
I20260812 09:48:20.514245 8229 raft_consensus.cc:348] T 136aadace1724175a669350f88ff6dbd P f9e2dce914b843b2bfc0de5f215a45ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9e2dce914b843b2bfc0de5f215a45ff" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 34429 } } peers { permanent_uuid: "6440e49253344afebefac34f6e39e878" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 40441 } } peers { permanent_uuid: "90b746efe2d54317b04ae5164cef1a3b" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 43737 } }
I20260812 09:48:20.514535 8229 raft_consensus.cc:374] T 136aadace1724175a669350f88ff6dbd P f9e2dce914b843b2bfc0de5f215a45ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:48:20.514601 8229 raft_consensus.cc:729] T 136aadace1724175a669350f88ff6dbd P f9e2dce914b843b2bfc0de5f215a45ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9e2dce914b843b2bfc0de5f215a45ff, State: Initialized, Role: FOLLOWER
I20260812 09:48:20.515033 8229 consensus_queue.cc:260] T 136aadace1724175a669350f88ff6dbd P f9e2dce914b843b2bfc0de5f215a45ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9e2dce914b843b2bfc0de5f215a45ff" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 34429 } } peers { permanent_uuid: "6440e49253344afebefac34f6e39e878" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 40441 } } peers { permanent_uuid: "90b746efe2d54317b04ae5164cef1a3b" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 43737 } }
I20260812 09:48:20.515628 7697 heartbeater.cc:503] Master 127.13.99.62:37085 was elected leader, sending a full tablet report...
I20260812 09:48:20.515971 8229 ts_tablet_manager.cc:1428] T 136aadace1724175a669350f88ff6dbd P f9e2dce914b843b2bfc0de5f215a45ff: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260812 09:48:20.516274 8234 tablet_bootstrap.cc:492] T 136aadace1724175a669350f88ff6dbd P 90b746efe2d54317b04ae5164cef1a3b: No bootstrap required, opened a new log
I20260812 09:48:20.516549 8234 ts_tablet_manager.cc:1397] T 136aadace1724175a669350f88ff6dbd P 90b746efe2d54317b04ae5164cef1a3b: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260812 09:48:20.519729 8234 raft_consensus.cc:348] T 136aadace1724175a669350f88ff6dbd P 90b746efe2d54317b04ae5164cef1a3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9e2dce914b843b2bfc0de5f215a45ff" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 34429 } } peers { permanent_uuid: "6440e49253344afebefac34f6e39e878" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 40441 } } peers { permanent_uuid: "90b746efe2d54317b04ae5164cef1a3b" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 43737 } }
I20260812 09:48:20.519928 8234 raft_consensus.cc:374] T 136aadace1724175a669350f88ff6dbd P 90b746efe2d54317b04ae5164cef1a3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:48:20.519960 8234 raft_consensus.cc:729] T 136aadace1724175a669350f88ff6dbd P 90b746efe2d54317b04ae5164cef1a3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90b746efe2d54317b04ae5164cef1a3b, State: Initialized, Role: FOLLOWER
I20260812 09:48:20.520269 8234 consensus_queue.cc:260] T 136aadace1724175a669350f88ff6dbd P 90b746efe2d54317b04ae5164cef1a3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9e2dce914b843b2bfc0de5f215a45ff" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 34429 } } peers { permanent_uuid: "6440e49253344afebefac34f6e39e878" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 40441 } } peers { permanent_uuid: "90b746efe2d54317b04ae5164cef1a3b" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 43737 } }
I20260812 09:48:20.520857 7954 heartbeater.cc:503] Master 127.13.99.62:37085 was elected leader, sending a full tablet report...
I20260812 09:48:20.521044 8234 ts_tablet_manager.cc:1428] T 136aadace1724175a669350f88ff6dbd P 90b746efe2d54317b04ae5164cef1a3b: Time spent starting tablet: real 0.004s user 0.001s sys 0.003s
I20260812 09:48:20.643913 8240 raft_consensus.cc:482] T 136aadace1724175a669350f88ff6dbd P f9e2dce914b843b2bfc0de5f215a45ff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 09:48:20.644189 8240 raft_consensus.cc:504] T 136aadace1724175a669350f88ff6dbd P f9e2dce914b843b2bfc0de5f215a45ff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9e2dce914b843b2bfc0de5f215a45ff" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 34429 } } peers { permanent_uuid: "6440e49253344afebefac34f6e39e878" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 40441 } } peers { permanent_uuid: "90b746efe2d54317b04ae5164cef1a3b" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 43737 } }
I20260812 09:48:20.645470 8240 leader_election.cc:288] T 136aadace1724175a669350f88ff6dbd P f9e2dce914b843b2bfc0de5f215a45ff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6440e49253344afebefac34f6e39e878 (127.13.99.3:40441), 90b746efe2d54317b04ae5164cef1a3b (127.13.99.2:43737)
I20260812 09:48:20.649719 8219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:20.645847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.1:40691 (local address 127.13.99.3:40441)
0812 09:48:20.646008 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0812 09:48:20.646014 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 09:48:20.646227 (+ 213us) server_negotiation.cc:407] Connection header received
0812 09:48:20.646324 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:48:20.646328 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:48:20.646384 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:48:20.646473 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:48:20.646848 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:48:20.647396 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:48:20.648681 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:48:20.649086 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:48:20.649222 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:48:20.649284 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0812 09:48:20.649510 (+ 226us) server_negotiation.cc:299] Negotiation successful
0812 09:48:20.649592 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260812 09:48:20.649734 8226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:20.645870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.1:35187 (local address 127.13.99.2:43737)
0812 09:48:20.646077 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0812 09:48:20.646082 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:48:20.646330 (+ 248us) server_negotiation.cc:407] Connection header received
0812 09:48:20.646477 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:48:20.646481 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:48:20.646541 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:48:20.646624 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:48:20.646934 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:48:20.647417 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:48:20.648658 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:48:20.649116 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:48:20.649222 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:48:20.649325 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0812 09:48:20.649561 (+ 236us) server_negotiation.cc:299] Negotiation successful
0812 09:48:20.649630 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260812 09:48:20.649713 8244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:20.645706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.3:40441 (local address 127.13.99.1:40691)
0812 09:48:20.646098 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0812 09:48:20.646115 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 09:48:20.646235 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:48:20.646495 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:48:20.646498 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:48:20.646516 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:48:20.646676 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:48:20.646684 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:48:20.647530 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:48:20.647534 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:48:20.648523 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:48:20.648538 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:48:20.649228 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:48:20.649231 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:48:20.649345 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:48:20.649370 (+ 25us) client_negotiation.cc:769] Sending connection context
0812 09:48:20.649450 (+ 80us) client_negotiation.cc:240] Negotiation successful
0812 09:48:20.649530 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"mutex_wait_us":6,"thread_start_us":133,"threads_started":1}
I20260812 09:48:20.649713 8245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:20.645743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.2:43737 (local address 127.13.99.1:35187)
0812 09:48:20.646206 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0812 09:48:20.646223 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 09:48:20.646348 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:48:20.646657 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:48:20.646660 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:48:20.646675 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:48:20.646813 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:48:20.646821 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:48:20.647570 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:48:20.647573 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:48:20.648486 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:48:20.648521 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:48:20.649359 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:48:20.649363 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:48:20.649463 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:48:20.649482 (+ 19us) client_negotiation.cc:769] Sending connection context
0812 09:48:20.649546 (+ 64us) client_negotiation.cc:240] Negotiation successful
0812 09:48:20.649597 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"mutex_wait_us":51,"thread_start_us":122,"threads_started":1}
I20260812 09:48:20.650430 7873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "136aadace1724175a669350f88ff6dbd" candidate_uuid: "f9e2dce914b843b2bfc0de5f215a45ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90b746efe2d54317b04ae5164cef1a3b" is_pre_election: true
I20260812 09:48:20.650563 8104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "136aadace1724175a669350f88ff6dbd" candidate_uuid: "f9e2dce914b843b2bfc0de5f215a45ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6440e49253344afebefac34f6e39e878" is_pre_election: true
I20260812 09:48:20.650791 7873 raft_consensus.cc:2454] T 136aadace1724175a669350f88ff6dbd P 90b746efe2d54317b04ae5164cef1a3b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9e2dce914b843b2bfc0de5f215a45ff in term 0.
I20260812 09:48:20.650950 8104 raft_consensus.cc:2454] T 136aadace1724175a669350f88ff6dbd P 6440e49253344afebefac34f6e39e878 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9e2dce914b843b2bfc0de5f215a45ff in term 0.
I20260812 09:48:20.651336 7503 leader_election.cc:302] T 136aadace1724175a669350f88ff6dbd P f9e2dce914b843b2bfc0de5f215a45ff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90b746efe2d54317b04ae5164cef1a3b, f9e2dce914b843b2bfc0de5f215a45ff; no voters:
I20260812 09:48:20.651650 8240 raft_consensus.cc:2781] T 136aadace1724175a669350f88ff6dbd P f9e2dce914b843b2bfc0de5f215a45ff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 09:48:20.651743 8240 raft_consensus.cc:482] T 136aadace1724175a669350f88ff6dbd P f9e2dce914b843b2bfc0de5f215a45ff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 09:48:20.651789 8240 raft_consensus.cc:3037] T 136aadace1724175a669350f88ff6dbd P f9e2dce914b843b2bfc0de5f215a45ff [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:48:20.653075 8240 raft_consensus.cc:504] T 136aadace1724175a669350f88ff6dbd P f9e2dce914b843b2bfc0de5f215a45ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9e2dce914b843b2bfc0de5f215a45ff" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 34429 } } peers { permanent_uuid: "6440e49253344afebefac34f6e39e878" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 40441 } } peers { permanent_uuid: "90b746efe2d54317b04ae5164cef1a3b" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 43737 } }
I20260812 09:48:20.653421 8240 leader_election.cc:288] T 136aadace1724175a669350f88ff6dbd P f9e2dce914b843b2bfc0de5f215a45ff [CANDIDATE]: Term 1 election: Requested vote from peers 6440e49253344afebefac34f6e39e878 (127.13.99.3:40441), 90b746efe2d54317b04ae5164cef1a3b (127.13.99.2:43737)
I20260812 09:48:20.653759 7873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "136aadace1724175a669350f88ff6dbd" candidate_uuid: "f9e2dce914b843b2bfc0de5f215a45ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90b746efe2d54317b04ae5164cef1a3b"
I20260812 09:48:20.653788 8104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "136aadace1724175a669350f88ff6dbd" candidate_uuid: "f9e2dce914b843b2bfc0de5f215a45ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6440e49253344afebefac34f6e39e878"
I20260812 09:48:20.653899 7873 raft_consensus.cc:3037] T 136aadace1724175a669350f88ff6dbd P 90b746efe2d54317b04ae5164cef1a3b [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:48:20.653957 8104 raft_consensus.cc:3037] T 136aadace1724175a669350f88ff6dbd P 6440e49253344afebefac34f6e39e878 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:48:20.655264 8104 raft_consensus.cc:2454] T 136aadace1724175a669350f88ff6dbd P 6440e49253344afebefac34f6e39e878 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9e2dce914b843b2bfc0de5f215a45ff in term 1.
I20260812 09:48:20.655335 7873 raft_consensus.cc:2454] T 136aadace1724175a669350f88ff6dbd P 90b746efe2d54317b04ae5164cef1a3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9e2dce914b843b2bfc0de5f215a45ff in term 1.
I20260812 09:48:20.655620 7503 leader_election.cc:302] T 136aadace1724175a669350f88ff6dbd P f9e2dce914b843b2bfc0de5f215a45ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90b746efe2d54317b04ae5164cef1a3b, f9e2dce914b843b2bfc0de5f215a45ff; no voters:
I20260812 09:48:20.655800 8240 raft_consensus.cc:2781] T 136aadace1724175a669350f88ff6dbd P f9e2dce914b843b2bfc0de5f215a45ff [term 1 FOLLOWER]: Leader election won for term 1
I20260812 09:48:20.656025 8240 raft_consensus.cc:686] T 136aadace1724175a669350f88ff6dbd P f9e2dce914b843b2bfc0de5f215a45ff [term 1 LEADER]: Becoming Leader. State: Replica: f9e2dce914b843b2bfc0de5f215a45ff, State: Running, Role: LEADER
I20260812 09:48:20.656389 8240 consensus_queue.cc:237] T 136aadace1724175a669350f88ff6dbd P f9e2dce914b843b2bfc0de5f215a45ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9e2dce914b843b2bfc0de5f215a45ff" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 34429 } } peers { permanent_uuid: "6440e49253344afebefac34f6e39e878" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 40441 } } peers { permanent_uuid: "90b746efe2d54317b04ae5164cef1a3b" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 43737 } }
I20260812 09:48:20.659654 7349 catalog_manager.cc:5495] T 136aadace1724175a669350f88ff6dbd P f9e2dce914b843b2bfc0de5f215a45ff reported cstate change: term changed from 0 to 1, leader changed from <none> to f9e2dce914b843b2bfc0de5f215a45ff (127.13.99.1). New cstate: current_term: 1 leader_uuid: "f9e2dce914b843b2bfc0de5f215a45ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9e2dce914b843b2bfc0de5f215a45ff" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 34429 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6440e49253344afebefac34f6e39e878" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 40441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90b746efe2d54317b04ae5164cef1a3b" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 43737 } health_report { overall_health: UNKNOWN } } }
W20260812 09:48:20.682864 8197 tablet.cc:2369] T 136aadace1724175a669350f88ff6dbd P 6440e49253344afebefac34f6e39e878: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 09:48:20.684742 8223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:20.680949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42192 (local address 127.13.99.1:34429)
0812 09:48:20.681118 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0812 09:48:20.681121 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 09:48:20.681136 (+ 15us) server_negotiation.cc:407] Connection header received
0812 09:48:20.681219 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:48:20.681221 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:48:20.681280 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:48:20.681376 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 09:48:20.681618 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:48:20.682124 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:48:20.683222 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:48:20.683453 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:48:20.683548 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:48:20.683899 (+ 351us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 09:48:20.684329 (+ 430us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 09:48:20.684443 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0812 09:48:20.684575 (+ 132us) server_negotiation.cc:299] Negotiation successful
0812 09:48:20.684635 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260812 09:48:20.690948 8104 raft_consensus.cc:1261] T 136aadace1724175a669350f88ff6dbd P 6440e49253344afebefac34f6e39e878 [term 1 FOLLOWER]: Refusing update from remote peer f9e2dce914b843b2bfc0de5f215a45ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 09:48:20.690948 7873 raft_consensus.cc:1261] T 136aadace1724175a669350f88ff6dbd P 90b746efe2d54317b04ae5164cef1a3b [term 1 FOLLOWER]: Refusing update from remote peer f9e2dce914b843b2bfc0de5f215a45ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 09:48:20.691690 8246 consensus_queue.cc:1035] T 136aadace1724175a669350f88ff6dbd P f9e2dce914b843b2bfc0de5f215a45ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "6440e49253344afebefac34f6e39e878" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 40441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 09:48:20.691915 8240 consensus_queue.cc:1035] T 136aadace1724175a669350f88ff6dbd P f9e2dce914b843b2bfc0de5f215a45ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "90b746efe2d54317b04ae5164cef1a3b" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 43737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 09:48:20.701047 8252 mvcc.cc:205] Tried to move back new op lower bound from 7317619100422500352 to 7317619100291862528. Current Snapshot: MvccSnapshot[applied={T|T < 7317619100422500352}]
I20260812 09:48:20.709962 7349 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39594:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 09:48:20.710119 7349 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39594:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 09:48:20.712532 7349 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6dbe64111bfb4d4a8730f47f3a1a4fb0: Sending DeleteTablet for 3 replicas of tablet 136aadace1724175a669350f88ff6dbd
I20260812 09:48:20.713333 7577 tablet_service.cc:1514] Processing DeleteTablet for tablet 136aadace1724175a669350f88ff6dbd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 09:48:20 UTC) from {username='kudu', principal='kudu/127.13.99.62@KRBTEST.COM'} at 127.0.0.1:42188
I20260812 09:48:20.713652 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 7478
I20260812 09:48:20.713845 8066 tablet_service.cc:1514] Processing DeleteTablet for tablet 136aadace1724175a669350f88ff6dbd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 09:48:20 UTC) from {username='kudu', principal='kudu/127.13.99.62@KRBTEST.COM'} at 127.0.0.1:53146
I20260812 09:48:20.716811 7833 tablet_service.cc:1514] Processing DeleteTablet for tablet 136aadace1724175a669350f88ff6dbd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 09:48:20 UTC) from {username='kudu', principal='kudu/127.13.99.62@KRBTEST.COM'} at 127.0.0.1:37254
I20260812 09:48:20.719822 8275 tablet_replica.cc:330] T 136aadace1724175a669350f88ff6dbd P 90b746efe2d54317b04ae5164cef1a3b: stopping tablet replica
I20260812 09:48:20.720206 8275 raft_consensus.cc:2229] T 136aadace1724175a669350f88ff6dbd P 90b746efe2d54317b04ae5164cef1a3b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 09:48:20.720408 8274 tablet_replica.cc:330] T 136aadace1724175a669350f88ff6dbd P 6440e49253344afebefac34f6e39e878: stopping tablet replica
I20260812 09:48:20.720594 8275 raft_consensus.cc:2258] T 136aadace1724175a669350f88ff6dbd P 90b746efe2d54317b04ae5164cef1a3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 09:48:20.720727 8274 raft_consensus.cc:2229] T 136aadace1724175a669350f88ff6dbd P 6440e49253344afebefac34f6e39e878 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 09:48:20.721014 8274 raft_consensus.cc:2258] T 136aadace1724175a669350f88ff6dbd P 6440e49253344afebefac34f6e39e878 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 09:48:20.723352 8274 ts_tablet_manager.cc:1905] T 136aadace1724175a669350f88ff6dbd P 6440e49253344afebefac34f6e39e878: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 09:48:20.723680 8275 ts_tablet_manager.cc:1905] T 136aadace1724175a669350f88ff6dbd P 90b746efe2d54317b04ae5164cef1a3b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 09:48:20.725770 8274 ts_tablet_manager.cc:1918] T 136aadace1724175a669350f88ff6dbd P 6440e49253344afebefac34f6e39e878: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 09:48:20.725888 8274 log.cc:1196] T 136aadace1724175a669350f88ff6dbd P 6440e49253344afebefac34f6e39e878: Deleting WAL directory at /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-2/wal/wals/136aadace1724175a669350f88ff6dbd
I20260812 09:48:20.726267 8274 ts_tablet_manager.cc:1939] T 136aadace1724175a669350f88ff6dbd P 6440e49253344afebefac34f6e39e878: Deleting consensus metadata
I20260812 09:48:20.727138 8275 ts_tablet_manager.cc:1918] T 136aadace1724175a669350f88ff6dbd P 90b746efe2d54317b04ae5164cef1a3b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 09:48:20.727258 8275 log.cc:1196] T 136aadace1724175a669350f88ff6dbd P 90b746efe2d54317b04ae5164cef1a3b: Deleting WAL directory at /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-1/wal/wals/136aadace1724175a669350f88ff6dbd
I20260812 09:48:20.727476 7333 catalog_manager.cc:4841] TS 6440e49253344afebefac34f6e39e878 (127.13.99.3:40441): tablet 136aadace1724175a669350f88ff6dbd (table test-table [id=c569756a8c664219a2b8f04917450b68]) successfully deleted
I20260812 09:48:20.727599 8275 ts_tablet_manager.cc:1939] T 136aadace1724175a669350f88ff6dbd P 90b746efe2d54317b04ae5164cef1a3b: Deleting consensus metadata
I20260812 09:48:20.728909 7332 catalog_manager.cc:4841] TS 90b746efe2d54317b04ae5164cef1a3b (127.13.99.2:43737): tablet 136aadace1724175a669350f88ff6dbd (table test-table [id=c569756a8c664219a2b8f04917450b68]) successfully deleted
W20260812 09:48:20.730499 7334 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.99.1:34429 (error 108)
I20260812 09:48:20.730857 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 7706
I20260812 09:48:20.733851 8218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:20.733480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.1:34429 (local address 127.0.0.1:42206)
0812 09:48:20.733656 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0812 09:48:20.733744 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.1:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260812 09:48:20.734040 7334 catalog_manager.cc:4584] TS f9e2dce914b843b2bfc0de5f215a45ff (127.13.99.1:34429): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 136aadace1724175a669350f88ff6dbd: Network error: Client connection negotiation failed: client connection to 127.13.99.1:34429: connect: Connection refused (error 111)
I20260812 09:48:20.745196 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 7961
I20260812 09:48:20.762225 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 7308
I20260812 09:48:40.779306 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.62:37085
--webserver_interface=127.13.99.62
--webserver_port=34893
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab
--principal=kudu/127.13.99.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:39987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.99.62:37085
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:48:40.903836 8283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:48:40.904089 8283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:48:40.904129 8283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:48:40.908222 8283 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260812 09:48:40.908277 8283 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260812 09:48:40.908293 8283 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260812 09:48:40.908306 8283 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 09:48:40.908324 8283 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:48:40.908373 8283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:48:40.908413 8283 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 09:48:40.908438 8283 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 09:48:40.913775 8283 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:39987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.13.99.62:37085
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab
--principal=kudu/127.13.99.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.62:37085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.99.62
--webserver_port=34893
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8283
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:48:40.914873 8283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:48:40.916000 8283 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:48:40.930687 8302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:48:40.930687 8295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:48:40.930838 8283 server_base.cc:1034] running on GCE node
W20260812 09:48:40.931010 8296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:48:40.931676 8283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:48:40.932981 8283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:48:40.934183 8283 hybrid_clock.cc:648] HybridClock initialized: now 1786528120934157 us; error 41 us; skew 500 ppm
Aug 12 09:48:40 dist-test-slave-jq4t krb5kdc[7284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528120, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:48:40.944759 8283 init.cc:377] Logged in from keytab as kudu/127.13.99.62@KRBTEST.COM (short username kudu)
I20260812 09:48:40.946193 8283 webserver.cc:460] Webserver started at http://127.13.99.62:34893/ using document root <none> and password file <none>
I20260812 09:48:40.946851 8283 fs_manager.cc:359] Metadata directory not provided
I20260812 09:48:40.946964 8283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:48:40.951216 8283 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.000s
I20260812 09:48:40.953629 8317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:48:40.954910 8283 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 09:48:40.955075 8283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/master-0/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/master-0/wal
uuid: "6dbe64111bfb4d4a8730f47f3a1a4fb0"
format_stamp: "Formatted at 2026-08-12 09:48:19 on dist-test-slave-jq4t"
server_key: "e48ace4008e2743289311f3907fe64d7"
server_key_iv: "935ea577c777a918b942eb11a9b78c31"
server_key_version: "encryptionkey@0"
I20260812 09:48:40.955714 8283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:48:40.972373 8283 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:48:40.975621 8283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:48:40.975862 8283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:48:40.983662 8283 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.62:37085
I20260812 09:48:40.984014 8428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.62:37085 every 8 connection(s)
I20260812 09:48:40.984866 8283 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
I20260812 09:48:40.988771 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 8283
I20260812 09:48:40.989248 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.99.1:34429
--local_ip_for_outbound_sockets=127.13.99.1
--tserver_master_addrs=127.13.99.62:37085
--webserver_port=36855
--webserver_interface=127.13.99.1
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.1.keytab
--principal=kudu/127.13.99.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:39987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 09:48:40.989826 8431 sys_catalog.cc:263] Verifying existing consensus state
I20260812 09:48:40.991631 8431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6dbe64111bfb4d4a8730f47f3a1a4fb0: Bootstrap starting.
I20260812 09:48:41.010934 8431 log.cc:824] T 00000000000000000000000000000000 P 6dbe64111bfb4d4a8730f47f3a1a4fb0: Log is configured to *not* fsync() on all Append() calls
I20260812 09:48:41.020044 8431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6dbe64111bfb4d4a8730f47f3a1a4fb0: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 09:48:41.020460 8431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6dbe64111bfb4d4a8730f47f3a1a4fb0: Bootstrap complete.
I20260812 09:48:41.024866 8431 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6dbe64111bfb4d4a8730f47f3a1a4fb0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dbe64111bfb4d4a8730f47f3a1a4fb0" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 37085 } }
I20260812 09:48:41.025309 8431 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6dbe64111bfb4d4a8730f47f3a1a4fb0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dbe64111bfb4d4a8730f47f3a1a4fb0, State: Initialized, Role: FOLLOWER
I20260812 09:48:41.025815 8431 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6dbe64111bfb4d4a8730f47f3a1a4fb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dbe64111bfb4d4a8730f47f3a1a4fb0" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 37085 } }
I20260812 09:48:41.025982 8431 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6dbe64111bfb4d4a8730f47f3a1a4fb0 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 09:48:41.026073 8431 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6dbe64111bfb4d4a8730f47f3a1a4fb0 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 09:48:41.026201 8431 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6dbe64111bfb4d4a8730f47f3a1a4fb0 [term 1 FOLLOWER]: Advancing to term 2
I20260812 09:48:41.028177 8431 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6dbe64111bfb4d4a8730f47f3a1a4fb0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dbe64111bfb4d4a8730f47f3a1a4fb0" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 37085 } }
I20260812 09:48:41.028594 8431 leader_election.cc:302] T 00000000000000000000000000000000 P 6dbe64111bfb4d4a8730f47f3a1a4fb0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6dbe64111bfb4d4a8730f47f3a1a4fb0; no voters:
I20260812 09:48:41.028970 8431 leader_election.cc:288] T 00000000000000000000000000000000 P 6dbe64111bfb4d4a8730f47f3a1a4fb0 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260812 09:48:41.029058 8438 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6dbe64111bfb4d4a8730f47f3a1a4fb0 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 09:48:41.029296 8438 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6dbe64111bfb4d4a8730f47f3a1a4fb0 [term 2 LEADER]: Becoming Leader. State: Replica: 6dbe64111bfb4d4a8730f47f3a1a4fb0, State: Running, Role: LEADER
I20260812 09:48:41.029681 8438 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6dbe64111bfb4d4a8730f47f3a1a4fb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dbe64111bfb4d4a8730f47f3a1a4fb0" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 37085 } }
I20260812 09:48:41.030299 8431 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6dbe64111bfb4d4a8730f47f3a1a4fb0 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 09:48:41.031273 8440 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dbe64111bfb4d4a8730f47f3a1a4fb0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "6dbe64111bfb4d4a8730f47f3a1a4fb0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dbe64111bfb4d4a8730f47f3a1a4fb0" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 37085 } } }
I20260812 09:48:41.031421 8440 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dbe64111bfb4d4a8730f47f3a1a4fb0 [sys.catalog]: This master's current role is: LEADER
I20260812 09:48:41.031715 8441 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dbe64111bfb4d4a8730f47f3a1a4fb0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6dbe64111bfb4d4a8730f47f3a1a4fb0. Latest consensus state: current_term: 2 leader_uuid: "6dbe64111bfb4d4a8730f47f3a1a4fb0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dbe64111bfb4d4a8730f47f3a1a4fb0" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 37085 } } }
I20260812 09:48:41.031791 8441 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dbe64111bfb4d4a8730f47f3a1a4fb0 [sys.catalog]: This master's current role is: LEADER
I20260812 09:48:41.031852 8450 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 09:48:41.036955 8450 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 09:48:41.037662 8450 catalog_manager.cc:1259] Loaded cluster ID: 68a068ddd80a4047bf4e04d78077accc
I20260812 09:48:41.037730 8450 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 09:48:41.040297 8450 catalog_manager.cc:1504] Loading token signing keys...
I20260812 09:48:41.041071 8450 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6dbe64111bfb4d4a8730f47f3a1a4fb0: Loaded TSK: 0
I20260812 09:48:41.056547 8450 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6dbe64111bfb4d4a8730f47f3a1a4fb0: Generated new TSK 1
I20260812 09:48:41.056808 8450 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 09:48:41.111907 8436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:48:41.112216 8436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:48:41.112299 8436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:48:41.116668 8436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:48:41.116747 8436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:48:41.116837 8436 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.99.1
I20260812 09:48:41.122098 8436 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:39987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.1.keytab
--principal=kudu/127.13.99.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.99.1:34429
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.99.1
--webserver_port=36855
--tserver_master_addrs=127.13.99.62:37085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8436
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.99.1
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:48:41.123143 8436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:48:41.124171 8436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:48:41.135339 8479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:48:41.135329 8481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:48:41.135811 8483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:48:41.135910 8436 server_base.cc:1034] running on GCE node
I20260812 09:48:41.136334 8436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:48:41.137053 8436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:48:41.138273 8436 hybrid_clock.cc:648] HybridClock initialized: now 1786528121138241 us; error 49 us; skew 500 ppm
Aug 12 09:48:41 dist-test-slave-jq4t krb5kdc[7284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528121, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:48:41.141390 8436 init.cc:377] Logged in from keytab as kudu/127.13.99.1@KRBTEST.COM (short username kudu)
I20260812 09:48:41.142427 8436 webserver.cc:460] Webserver started at http://127.13.99.1:36855/ using document root <none> and password file <none>
I20260812 09:48:41.143033 8436 fs_manager.cc:359] Metadata directory not provided
I20260812 09:48:41.143116 8436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:48:41.147212 8436 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260812 09:48:41.149407 8494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:48:41.150710 8436 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260812 09:48:41.150852 8436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-0/wal
uuid: "f9e2dce914b843b2bfc0de5f215a45ff"
format_stamp: "Formatted at 2026-08-12 09:48:19 on dist-test-slave-jq4t"
server_key: "493d29e9ebcdf3ef9dadeb75241b9724"
server_key_iv: "322088c3e4178e5fd7efe60477020e3b"
server_key_version: "encryptionkey@0"
I20260812 09:48:41.151398 8436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:48:41.169013 8436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:48:41.172724 8436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:48:41.172935 8436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:48:41.173604 8436 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 09:48:41.174837 8506 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260812 09:48:41.179247 8436 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 09:48:41.179313 8436 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260812 09:48:41.179368 8436 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260812 09:48:41.181165 8436 ts_tablet_manager.cc:610] Registered 1 tablets
I20260812 09:48:41.181226 8436 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260812 09:48:41.181493 8506 tablet_bootstrap.cc:492] T 136aadace1724175a669350f88ff6dbd P f9e2dce914b843b2bfc0de5f215a45ff: Bootstrap starting.
I20260812 09:48:41.192766 8436 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.1:34429
I20260812 09:48:41.192782 8697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.1:34429 every 8 connection(s)
I20260812 09:48:41.194036 8436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-0/data/info.pb
I20260812 09:48:41.197646 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 8436
I20260812 09:48:41.198158 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.99.2:43737
--local_ip_for_outbound_sockets=127.13.99.2
--tserver_master_addrs=127.13.99.62:37085
--webserver_port=37537
--webserver_interface=127.13.99.2
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.2.keytab
--principal=kudu/127.13.99.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:39987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 09:48:41.203588 8506 log.cc:824] T 136aadace1724175a669350f88ff6dbd P f9e2dce914b843b2bfc0de5f215a45ff: Log is configured to *not* fsync() on all Append() calls
Aug 12 09:48:41 dist-test-slave-jq4t krb5kdc[7284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528121, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.1@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
I20260812 09:48:41.212764 8506 tablet_bootstrap.cc:492] T 136aadace1724175a669350f88ff6dbd P f9e2dce914b843b2bfc0de5f215a45ff: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 09:48:41.213267 8506 tablet_bootstrap.cc:492] T 136aadace1724175a669350f88ff6dbd P f9e2dce914b843b2bfc0de5f215a45ff: Bootstrap complete.
I20260812 09:48:41.214664 8702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:41.196290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.1:37225 (local address 127.13.99.62:37085)
0812 09:48:41.196687 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0812 09:48:41.196704 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 09:48:41.197123 (+ 419us) server_negotiation.cc:407] Connection header received
0812 09:48:41.198679 (+ 1556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:48:41.198698 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:48:41.199278 (+ 580us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:48:41.199628 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:48:41.201394 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:48:41.202392 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:48:41.203613 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:48:41.204263 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:48:41.204454 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:48:41.207947 (+ 3493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:48:41.207954 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:48:41.207972 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:48:41.208008 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:48:41.210333 (+ 2325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:48:41.211122 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:48:41.211129 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:48:41.211134 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:48:41.211302 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:48:41.211804 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:48:41.211807 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:48:41.211809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:48:41.212291 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:48:41.212441 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0812 09:48:41.213399 (+ 958us) server_negotiation.cc:299] Negotiation successful
0812 09:48:41.213740 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":135,"threads_started":1}
I20260812 09:48:41.214704 8703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:41.196305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:37085 (local address 127.13.99.1:37225)
0812 09:48:41.196953 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0812 09:48:41.196995 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 09:48:41.198136 (+ 1141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:48:41.200280 (+ 2144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:48:41.200287 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:48:41.200732 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:48:41.201201 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:48:41.201216 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:48:41.202555 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:48:41.202558 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:48:41.203473 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:48:41.203484 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:48:41.204444 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:48:41.204451 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:48:41.204736 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:48:41.205515 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:48:41.205554 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:48:41.207689 (+ 2135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:48:41.210494 (+ 2805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:48:41.210502 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:48:41.210518 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:48:41.210958 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:48:41.211436 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:48:41.211442 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:48:41.211446 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:48:41.211666 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:48:41.212451 (+ 785us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:48:41.212460 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:48:41.212856 (+ 396us) client_negotiation.cc:769] Sending connection context
0812 09:48:41.213198 (+ 342us) client_negotiation.cc:240] Negotiation successful
0812 09:48:41.213592 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":189,"threads_started":1}
I20260812 09:48:41.214748 8506 ts_tablet_manager.cc:1397] T 136aadace1724175a669350f88ff6dbd P f9e2dce914b843b2bfc0de5f215a45ff: Time spent bootstrapping tablet: real 0.033s user 0.022s sys 0.009s
I20260812 09:48:41.217713 8699 heartbeater.cc:347] Connected to a master server at 127.13.99.62:37085
I20260812 09:48:41.217965 8699 heartbeater.cc:464] Registering TS with master...
I20260812 09:48:41.217587 8506 raft_consensus.cc:348] T 136aadace1724175a669350f88ff6dbd P f9e2dce914b843b2bfc0de5f215a45ff [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9e2dce914b843b2bfc0de5f215a45ff" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 34429 } } peers { permanent_uuid: "6440e49253344afebefac34f6e39e878" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 40441 } } peers { permanent_uuid: "90b746efe2d54317b04ae5164cef1a3b" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 43737 } }
I20260812 09:48:41.218143 8506 raft_consensus.cc:729] T 136aadace1724175a669350f88ff6dbd P f9e2dce914b843b2bfc0de5f215a45ff [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9e2dce914b843b2bfc0de5f215a45ff, State: Initialized, Role: FOLLOWER
I20260812 09:48:41.218514 8699 heartbeater.cc:511] Master 127.13.99.62:37085 requested a full tablet report, sending...
I20260812 09:48:41.218712 8506 consensus_queue.cc:260] T 136aadace1724175a669350f88ff6dbd P f9e2dce914b843b2bfc0de5f215a45ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9e2dce914b843b2bfc0de5f215a45ff" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 34429 } } peers { permanent_uuid: "6440e49253344afebefac34f6e39e878" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 40441 } } peers { permanent_uuid: "90b746efe2d54317b04ae5164cef1a3b" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 43737 } }
I20260812 09:48:41.219821 8506 ts_tablet_manager.cc:1428] T 136aadace1724175a669350f88ff6dbd P f9e2dce914b843b2bfc0de5f215a45ff: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260812 09:48:41.221369 8352 ts_manager.cc:194] Registered new tserver with Master: f9e2dce914b843b2bfc0de5f215a45ff (127.13.99.1:34429)
I20260812 09:48:41.224810 8352 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.99.1@KRBTEST.COM'} at 127.13.99.1:37225
I20260812 09:48:41.227447 8699 heartbeater.cc:503] Master 127.13.99.62:37085 was elected leader, sending a full tablet report...
Aug 12 09:48:41 dist-test-slave-jq4t krb5kdc[7284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528120, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.62@KRBTEST.COM for kudu/127.13.99.1@KRBTEST.COM
I20260812 09:48:41.235497 8721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:41.224118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.1:34429 (local address 127.0.0.1:56532)
0812 09:48:41.224912 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0812 09:48:41.224959 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0812 09:48:41.225102 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:48:41.225629 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:48:41.225914 (+ 285us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:48:41.226342 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:48:41.226496 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:48:41.226508 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:48:41.227818 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:48:41.227822 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:48:41.228662 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:48:41.228673 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:48:41.229378 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:48:41.229381 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:48:41.229487 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:48:41.230072 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:48:41.230088 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:48:41.231661 (+ 1573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:48:41.233695 (+ 2034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:48:41.233706 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:48:41.233711 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:48:41.233974 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:48:41.234297 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:48:41.234300 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:48:41.234302 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:48:41.234392 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:48:41.234847 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:48:41.234853 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:48:41.234990 (+ 137us) client_negotiation.cc:769] Sending connection context
0812 09:48:41.235133 (+ 143us) client_negotiation.cc:240] Negotiation successful
0812 09:48:41.235323 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":102,"threads_started":1}
I20260812 09:48:41.235533 8722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:41.224225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56532 (local address 127.13.99.1:34429)
0812 09:48:41.224633 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0812 09:48:41.224645 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 09:48:41.225063 (+ 418us) server_negotiation.cc:407] Connection header received
0812 09:48:41.225252 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:48:41.225257 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:48:41.225459 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:48:41.225608 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:48:41.226640 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:48:41.227676 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:48:41.228787 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:48:41.229242 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:48:41.229343 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:48:41.231804 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:48:41.231822 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:48:41.231832 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:48:41.231853 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:48:41.233570 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:48:41.234111 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:48:41.234116 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:48:41.234117 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:48:41.234195 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:48:41.234489 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:48:41.234493 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:48:41.234494 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:48:41.234723 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:48:41.234886 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0812 09:48:41.235218 (+ 332us) server_negotiation.cc:299] Negotiation successful
0812 09:48:41.235389 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":111,"threads_started":1}
I20260812 09:48:41.236462 8579 tablet_service.cc:1514] Processing DeleteTablet for tablet 136aadace1724175a669350f88ff6dbd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 09:48:20 UTC) from {username='kudu', principal='kudu/127.13.99.62@KRBTEST.COM'} at 127.0.0.1:56532
I20260812 09:48:41.237443 8723 tablet_replica.cc:330] stopping tablet replica
I20260812 09:48:41.237738 8723 raft_consensus.cc:2229] T 136aadace1724175a669350f88ff6dbd P f9e2dce914b843b2bfc0de5f215a45ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 09:48:41.237990 8723 raft_consensus.cc:2258] T 136aadace1724175a669350f88ff6dbd P f9e2dce914b843b2bfc0de5f215a45ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 09:48:41.238906 8723 ts_tablet_manager.cc:1905] T 136aadace1724175a669350f88ff6dbd P f9e2dce914b843b2bfc0de5f215a45ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 09:48:41.242491 8723 ts_tablet_manager.cc:1918] T 136aadace1724175a669350f88ff6dbd P f9e2dce914b843b2bfc0de5f215a45ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 09:48:41.242607 8723 log.cc:1196] T 136aadace1724175a669350f88ff6dbd P f9e2dce914b843b2bfc0de5f215a45ff: Deleting WAL directory at /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-0/wal/wals/136aadace1724175a669350f88ff6dbd
I20260812 09:48:41.242892 8723 ts_tablet_manager.cc:1939] T 136aadace1724175a669350f88ff6dbd P f9e2dce914b843b2bfc0de5f215a45ff: Deleting consensus metadata
I20260812 09:48:41.244514 8322 catalog_manager.cc:4841] TS f9e2dce914b843b2bfc0de5f215a45ff (127.13.99.1:34429): tablet 136aadace1724175a669350f88ff6dbd (table test-table [id=c569756a8c664219a2b8f04917450b68]) successfully deleted
W20260812 09:48:41.320391 8708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:48:41.320652 8708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:48:41.320724 8708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:48:41.325343 8708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:48:41.325400 8708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:48:41.325482 8708 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.99.2
I20260812 09:48:41.331095 8708 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:39987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.2.keytab
--principal=kudu/127.13.99.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.99.2:43737
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.99.2
--webserver_port=37537
--tserver_master_addrs=127.13.99.62:37085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8708
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.99.2
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:48:41.332154 8708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:48:41.333252 8708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:48:41.342988 8734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:48:41.343034 8731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:48:41.343175 8732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:48:41.343112 8708 server_base.cc:1034] running on GCE node
I20260812 09:48:41.343623 8708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:48:41.344271 8708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:48:41.345458 8708 hybrid_clock.cc:648] HybridClock initialized: now 1786528121345421 us; error 50 us; skew 500 ppm
Aug 12 09:48:41 dist-test-slave-jq4t krb5kdc[7284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528121, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:48:41.348310 8708 init.cc:377] Logged in from keytab as kudu/127.13.99.2@KRBTEST.COM (short username kudu)
I20260812 09:48:41.349220 8708 webserver.cc:460] Webserver started at http://127.13.99.2:37537/ using document root <none> and password file <none>
I20260812 09:48:41.349794 8708 fs_manager.cc:359] Metadata directory not provided
I20260812 09:48:41.349912 8708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:48:41.353837 8708 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260812 09:48:41.356015 8744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:48:41.357394 8708 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260812 09:48:41.357545 8708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-1/wal
uuid: "90b746efe2d54317b04ae5164cef1a3b"
format_stamp: "Formatted at 2026-08-12 09:48:20 on dist-test-slave-jq4t"
server_key: "2304f1d32d8d1d9ae92fb41ef8950a57"
server_key_iv: "98ab55e79a37ce873231623ae8b1522c"
server_key_version: "encryptionkey@0"
I20260812 09:48:41.358080 8708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:48:41.370370 8708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:48:41.373643 8708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:48:41.373844 8708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:48:41.374421 8708 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 09:48:41.375258 8708 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:48:41.375334 8708 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 09:48:41.375423 8708 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:48:41.375471 8708 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 09:48:41.386986 8708 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.2:43737
I20260812 09:48:41.387168 8935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.2:43737 every 8 connection(s)
I20260812 09:48:41.388450 8708 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-1/data/info.pb
I20260812 09:48:41.389114 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 8708
I20260812 09:48:41.389689 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.99.3:40441
--local_ip_for_outbound_sockets=127.13.99.3
--tserver_master_addrs=127.13.99.62:37085
--webserver_port=34537
--webserver_interface=127.13.99.3
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.3.keytab
--principal=kudu/127.13.99.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:39987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 09:48:41 dist-test-slave-jq4t krb5kdc[7284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528121, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.2@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
I20260812 09:48:41.404410 8702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:41.390726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.2:32885 (local address 127.13.99.62:37085)
0812 09:48:41.391979 (+ 1253us) server_negotiation.cc:206] Beginning negotiation
0812 09:48:41.391984 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:48:41.392088 (+ 104us) server_negotiation.cc:407] Connection header received
0812 09:48:41.394046 (+ 1958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:48:41.394050 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:48:41.394115 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:48:41.394223 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:48:41.395450 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:48:41.395996 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:48:41.397262 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:48:41.397515 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:48:41.397636 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:48:41.400592 (+ 2956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:48:41.400596 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:48:41.400598 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:48:41.400626 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:48:41.402235 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:48:41.402802 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:48:41.402806 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:48:41.402807 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:48:41.402889 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:48:41.403245 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:48:41.403248 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:48:41.403250 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:48:41.403446 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:48:41.403561 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0812 09:48:41.404083 (+ 522us) server_negotiation.cc:299] Negotiation successful
0812 09:48:41.404228 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1135}
I20260812 09:48:41.405298 8945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:41.391220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:37085 (local address 127.13.99.2:32885)
0812 09:48:41.391908 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0812 09:48:41.391971 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0812 09:48:41.393806 (+ 1835us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:48:41.394418 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:48:41.394425 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:48:41.394782 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:48:41.395287 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:48:41.395301 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:48:41.396127 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:48:41.396130 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:48:41.397127 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:48:41.397137 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:48:41.397682 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:48:41.397686 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:48:41.397869 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:48:41.398569 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:48:41.398587 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:48:41.400376 (+ 1789us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:48:41.402371 (+ 1995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:48:41.402378 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:48:41.402393 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:48:41.402676 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:48:41.402981 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:48:41.402986 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:48:41.402991 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:48:41.403134 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:48:41.403561 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:48:41.403569 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:48:41.403829 (+ 260us) client_negotiation.cc:769] Sending connection context
0812 09:48:41.404025 (+ 196us) client_negotiation.cc:240] Negotiation successful
0812 09:48:41.404361 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":147,"threads_started":1}
I20260812 09:48:41.406503 8937 heartbeater.cc:347] Connected to a master server at 127.13.99.62:37085
I20260812 09:48:41.406772 8937 heartbeater.cc:464] Registering TS with master...
I20260812 09:48:41.407346 8937 heartbeater.cc:511] Master 127.13.99.62:37085 requested a full tablet report, sending...
I20260812 09:48:41.408454 8352 ts_manager.cc:194] Registered new tserver with Master: 90b746efe2d54317b04ae5164cef1a3b (127.13.99.2:43737)
I20260812 09:48:41.409103 8352 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.99.2@KRBTEST.COM'} at 127.13.99.2:32885
W20260812 09:48:41.511241 8949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:48:41.511512 8949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:48:41.511555 8949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:48:41.515702 8949 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:48:41.515769 8949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:48:41.515829 8949 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.99.3
I20260812 09:48:41.521159 8949 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:39987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.3.keytab
--principal=kudu/127.13.99.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.99.3:40441
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.99.3
--webserver_port=34537
--tserver_master_addrs=127.13.99.62:37085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8949
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.99.3
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:48:41.522337 8949 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:48:41.523396 8949 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:48:41.533595 8963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:48:41.533617 8966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:48:41.533617 8958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:48:41.533897 8949 server_base.cc:1034] running on GCE node
I20260812 09:48:41.534298 8949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:48:41.534938 8949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:48:41.536108 8949 hybrid_clock.cc:648] HybridClock initialized: now 1786528121536092 us; error 26 us; skew 500 ppm
Aug 12 09:48:41 dist-test-slave-jq4t krb5kdc[7284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528121, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:48:41.539026 8949 init.cc:377] Logged in from keytab as kudu/127.13.99.3@KRBTEST.COM (short username kudu)
I20260812 09:48:41.539952 8949 webserver.cc:460] Webserver started at http://127.13.99.3:34537/ using document root <none> and password file <none>
I20260812 09:48:41.540522 8949 fs_manager.cc:359] Metadata directory not provided
I20260812 09:48:41.540616 8949 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:48:41.544301 8949 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260812 09:48:41.546471 8975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:48:41.547744 8949 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 09:48:41.547849 8949 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-2/wal
uuid: "6440e49253344afebefac34f6e39e878"
format_stamp: "Formatted at 2026-08-12 09:48:20 on dist-test-slave-jq4t"
server_key: "523a68247cb62087c9374fbdf31e43b3"
server_key_iv: "c08b041a9e4a37ba82a2e3f18c15f5fb"
server_key_version: "encryptionkey@0"
I20260812 09:48:41.548404 8949 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:48:41.559149 8949 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:48:41.562688 8949 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:48:41.562880 8949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:48:41.563479 8949 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 09:48:41.564260 8949 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:48:41.564308 8949 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 09:48:41.564347 8949 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:48:41.564363 8949 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 09:48:41.573536 8949 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.3:40441
I20260812 09:48:41.573563 9174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.3:40441 every 8 connection(s)
I20260812 09:48:41.574527 8949 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-2/data/info.pb
I20260812 09:48:41.578568 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 8949
Aug 12 09:48:41 dist-test-slave-jq4t krb5kdc[7284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528121, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.3@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
I20260812 09:48:41.588171 9184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:41.579602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50978 (local address 127.13.99.62:37085)
0812 09:48:41.579899 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0812 09:48:41.579904 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:48:41.579918 (+ 14us) server_negotiation.cc:407] Connection header received
0812 09:48:41.579956 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:48:41.579959 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:48:41.580000 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:48:41.580096 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:48:41.580940 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:48:41.581463 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:48:41.582329 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:48:41.582618 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:48:41.582730 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:48:41.584229 (+ 1499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:48:41.584233 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:48:41.584237 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:48:41.584265 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:48:41.586254 (+ 1989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:48:41.586687 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:48:41.586692 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:48:41.586694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:48:41.586785 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:48:41.587071 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:48:41.587074 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:48:41.587076 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:48:41.587261 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:48:41.587483 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0812 09:48:41.587673 (+ 190us) server_negotiation.cc:299] Negotiation successful
0812 09:48:41.587800 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":219,"spinlock_wait_cycles":896,"thread_start_us":100,"threads_started":1}
I20260812 09:48:41.589564 8702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:41.576423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.3:40565 (local address 127.13.99.62:37085)
0812 09:48:41.576634 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0812 09:48:41.576638 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:48:41.577482 (+ 844us) server_negotiation.cc:407] Connection header received
0812 09:48:41.579116 (+ 1634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:48:41.579119 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:48:41.579180 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:48:41.579260 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:48:41.580647 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:48:41.581192 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:48:41.582345 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:48:41.582639 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:48:41.582769 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:48:41.585460 (+ 2691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:48:41.585470 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:48:41.585472 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:48:41.585495 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:48:41.586998 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:48:41.587560 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:48:41.587564 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:48:41.587565 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:48:41.587640 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:48:41.588182 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:48:41.588188 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:48:41.588193 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:48:41.588433 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:48:41.588562 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0812 09:48:41.589288 (+ 726us) server_negotiation.cc:299] Negotiation successful
0812 09:48:41.589410 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":109,"spinlock_wait_cycles":37120}
I20260812 09:48:41.590477 9179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:41.576832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:37085 (local address 127.13.99.3:40565)
0812 09:48:41.577323 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0812 09:48:41.577364 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0812 09:48:41.578866 (+ 1502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:48:41.579433 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:48:41.579440 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:48:41.579858 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:48:41.580462 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:48:41.580477 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:48:41.581319 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:48:41.581322 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:48:41.582216 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:48:41.582225 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:48:41.582795 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:48:41.582799 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:48:41.582993 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:48:41.583640 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:48:41.583657 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:48:41.585291 (+ 1634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:48:41.587135 (+ 1844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:48:41.587141 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:48:41.587156 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:48:41.587436 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:48:41.587738 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:48:41.587741 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:48:41.587743 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:48:41.587940 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:48:41.588621 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:48:41.588627 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:48:41.588958 (+ 331us) client_negotiation.cc:769] Sending connection context
0812 09:48:41.589175 (+ 217us) client_negotiation.cc:240] Negotiation successful
0812 09:48:41.589516 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":123,"threads_started":1}
I20260812 09:48:41.591719 9176 heartbeater.cc:347] Connected to a master server at 127.13.99.62:37085
I20260812 09:48:41.591965 9176 heartbeater.cc:464] Registering TS with master...
I20260812 09:48:41.592458 9176 heartbeater.cc:511] Master 127.13.99.62:37085 requested a full tablet report, sending...
I20260812 09:48:41.593534 8352 ts_manager.cc:194] Registered new tserver with Master: 6440e49253344afebefac34f6e39e878 (127.13.99.3:40441)
I20260812 09:48:41.594146 8352 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.99.3@KRBTEST.COM'} at 127.13.99.3:40565
I20260812 09:48:41.602110 13708 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 09:48:41.606483 8702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:41.602806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50988 (local address 127.13.99.62:37085)
0812 09:48:41.602965 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0812 09:48:41.602970 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:48:41.603142 (+ 172us) server_negotiation.cc:407] Connection header received
0812 09:48:41.603349 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:48:41.603352 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:48:41.603452 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:48:41.603576 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 09:48:41.603831 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:48:41.604337 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:48:41.605255 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:48:41.605476 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:48:41.605598 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:48:41.605921 (+ 323us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 09:48:41.606067 (+ 146us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0812 09:48:41.606343 (+ 276us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50988: token expired
Metrics: {"server-negotiator.queue_time_us":68}
W20260812 09:48:41.606576 8702 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50988: token expired
W20260812 09:48:41.606945 8207 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.13.99.62:37085)
I20260812 09:48:41.614648 8702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:41.607365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50992 (local address 127.13.99.62:37085)
0812 09:48:41.607481 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0812 09:48:41.607486 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:48:41.607525 (+ 39us) server_negotiation.cc:407] Connection header received
0812 09:48:41.607641 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:48:41.607644 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:48:41.607687 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:48:41.607771 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:48:41.608443 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:48:41.608996 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:48:41.609793 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:48:41.610007 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:48:41.610113 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:48:41.611495 (+ 1382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:48:41.611499 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:48:41.611502 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:48:41.611540 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:48:41.612989 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:48:41.613423 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:48:41.613427 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:48:41.613428 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:48:41.613505 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:48:41.613795 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:48:41.613799 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:48:41.613800 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:48:41.613980 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:48:41.614129 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0812 09:48:41.614366 (+ 237us) server_negotiation.cc:299] Negotiation successful
0812 09:48:41.614487 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260812 09:48:41.617347 8352 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50992:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 09:48:41.620589 8352 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 09:48:41.628193 8579 tablet_service.cc:1467] Processing CreateTablet for tablet 9e454f9dfb96467399fa4eb28c8ade3a (DEFAULT_TABLE table=test-table [id=3064472d0b14441a85372d4b7c4dc862]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 09:48:41.628954 8579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e454f9dfb96467399fa4eb28c8ade3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:48:41.632135 8506 tablet_bootstrap.cc:492] T 9e454f9dfb96467399fa4eb28c8ade3a P f9e2dce914b843b2bfc0de5f215a45ff: Bootstrap starting.
I20260812 09:48:41.632568 8721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:41.627074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.2:43737 (local address 127.0.0.1:56406)
0812 09:48:41.627281 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0812 09:48:41.627298 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 09:48:41.627377 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:48:41.628269 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:48:41.628273 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:48:41.628293 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:48:41.628405 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:48:41.628416 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:48:41.630036 (+ 1620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:48:41.630039 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:48:41.631139 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:48:41.631150 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:48:41.632114 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:48:41.632118 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:48:41.632220 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:48:41.632270 (+ 50us) client_negotiation.cc:769] Sending connection context
0812 09:48:41.632334 (+ 64us) client_negotiation.cc:240] Negotiation successful
0812 09:48:41.632413 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121}
I20260812 09:48:41.633160 8506 tablet_bootstrap.cc:654] T 9e454f9dfb96467399fa4eb28c8ade3a P f9e2dce914b843b2bfc0de5f215a45ff: Neither blocks nor log segments found. Creating new log.
I20260812 09:48:41.633361 9193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:41.627391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.3:40441 (local address 127.0.0.1:60694)
0812 09:48:41.627670 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0812 09:48:41.627684 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 09:48:41.627774 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:48:41.628770 (+ 996us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:48:41.628773 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:48:41.628793 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:48:41.628889 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:48:41.628896 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:48:41.630486 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:48:41.630489 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:48:41.631645 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:48:41.631657 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:48:41.632977 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:48:41.632981 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:48:41.633064 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:48:41.633085 (+ 21us) client_negotiation.cc:769] Sending connection context
0812 09:48:41.633150 (+ 65us) client_negotiation.cc:240] Negotiation successful
0812 09:48:41.633220 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":124,"threads_started":1}
I20260812 09:48:41.633482 9195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:41.627871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60694 (local address 127.13.99.3:40441)
0812 09:48:41.628223 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0812 09:48:41.628236 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 09:48:41.628257 (+ 21us) server_negotiation.cc:407] Connection header received
0812 09:48:41.628328 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:48:41.628333 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:48:41.628609 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:48:41.628771 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:48:41.629020 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:48:41.630380 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:48:41.631856 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:48:41.632875 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:48:41.632986 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:48:41.633156 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0812 09:48:41.633210 (+ 54us) server_negotiation.cc:299] Negotiation successful
0812 09:48:41.633316 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":162,"threads_started":1}
I20260812 09:48:41.634135 9192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:41.627265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56406 (local address 127.13.99.2:43737)
0812 09:48:41.627732 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0812 09:48:41.627746 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 09:48:41.627771 (+ 25us) server_negotiation.cc:407] Connection header received
0812 09:48:41.627858 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:48:41.627864 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:48:41.628106 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:48:41.628287 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:48:41.628587 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:48:41.629776 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:48:41.631276 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:48:41.631971 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:48:41.633192 (+ 1221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:48:41.633800 (+ 608us) server_negotiation.cc:1035] Waiting for connection context
0812 09:48:41.633863 (+ 63us) server_negotiation.cc:299] Negotiation successful
0812 09:48:41.633967 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":131,"threads_started":1}
I20260812 09:48:41.635044 8506 tablet_bootstrap.cc:492] T 9e454f9dfb96467399fa4eb28c8ade3a P f9e2dce914b843b2bfc0de5f215a45ff: No bootstrap required, opened a new log
I20260812 09:48:41.635143 8506 ts_tablet_manager.cc:1397] T 9e454f9dfb96467399fa4eb28c8ade3a P f9e2dce914b843b2bfc0de5f215a45ff: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 09:48:41.635597 8506 raft_consensus.cc:348] T 9e454f9dfb96467399fa4eb28c8ade3a P f9e2dce914b843b2bfc0de5f215a45ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b746efe2d54317b04ae5164cef1a3b" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 43737 } } peers { permanent_uuid: "6440e49253344afebefac34f6e39e878" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 40441 } } peers { permanent_uuid: "f9e2dce914b843b2bfc0de5f215a45ff" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 34429 } }
I20260812 09:48:41.635728 8506 raft_consensus.cc:374] T 9e454f9dfb96467399fa4eb28c8ade3a P f9e2dce914b843b2bfc0de5f215a45ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:48:41.635768 8506 raft_consensus.cc:729] T 9e454f9dfb96467399fa4eb28c8ade3a P f9e2dce914b843b2bfc0de5f215a45ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9e2dce914b843b2bfc0de5f215a45ff, State: Initialized, Role: FOLLOWER
I20260812 09:48:41.635957 8506 consensus_queue.cc:260] T 9e454f9dfb96467399fa4eb28c8ade3a P f9e2dce914b843b2bfc0de5f215a45ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b746efe2d54317b04ae5164cef1a3b" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 43737 } } peers { permanent_uuid: "6440e49253344afebefac34f6e39e878" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 40441 } } peers { permanent_uuid: "f9e2dce914b843b2bfc0de5f215a45ff" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 34429 } }
I20260812 09:48:41.636320 8506 ts_tablet_manager.cc:1428] T 9e454f9dfb96467399fa4eb28c8ade3a P f9e2dce914b843b2bfc0de5f215a45ff: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 09:48:41.636709 9067 tablet_service.cc:1467] Processing CreateTablet for tablet 9e454f9dfb96467399fa4eb28c8ade3a (DEFAULT_TABLE table=test-table [id=3064472d0b14441a85372d4b7c4dc862]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 09:48:41.637112 8825 tablet_service.cc:1467] Processing CreateTablet for tablet 9e454f9dfb96467399fa4eb28c8ade3a (DEFAULT_TABLE table=test-table [id=3064472d0b14441a85372d4b7c4dc862]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 09:48:41.637933 9067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e454f9dfb96467399fa4eb28c8ade3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:48:41.638581 8825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e454f9dfb96467399fa4eb28c8ade3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:48:41.644966 9198 tablet_bootstrap.cc:492] T 9e454f9dfb96467399fa4eb28c8ade3a P 6440e49253344afebefac34f6e39e878: Bootstrap starting.
I20260812 09:48:41.645222 9201 tablet_bootstrap.cc:492] T 9e454f9dfb96467399fa4eb28c8ade3a P 90b746efe2d54317b04ae5164cef1a3b: Bootstrap starting.
I20260812 09:48:41.647704 9198 tablet_bootstrap.cc:654] T 9e454f9dfb96467399fa4eb28c8ade3a P 6440e49253344afebefac34f6e39e878: Neither blocks nor log segments found. Creating new log.
I20260812 09:48:41.647733 9201 tablet_bootstrap.cc:654] T 9e454f9dfb96467399fa4eb28c8ade3a P 90b746efe2d54317b04ae5164cef1a3b: Neither blocks nor log segments found. Creating new log.
I20260812 09:48:41.648429 9198 log.cc:824] T 9e454f9dfb96467399fa4eb28c8ade3a P 6440e49253344afebefac34f6e39e878: Log is configured to *not* fsync() on all Append() calls
I20260812 09:48:41.648429 9201 log.cc:824] T 9e454f9dfb96467399fa4eb28c8ade3a P 90b746efe2d54317b04ae5164cef1a3b: Log is configured to *not* fsync() on all Append() calls
I20260812 09:48:41.650228 9201 tablet_bootstrap.cc:492] T 9e454f9dfb96467399fa4eb28c8ade3a P 90b746efe2d54317b04ae5164cef1a3b: No bootstrap required, opened a new log
I20260812 09:48:41.650236 9198 tablet_bootstrap.cc:492] T 9e454f9dfb96467399fa4eb28c8ade3a P 6440e49253344afebefac34f6e39e878: No bootstrap required, opened a new log
I20260812 09:48:41.650431 9201 ts_tablet_manager.cc:1397] T 9e454f9dfb96467399fa4eb28c8ade3a P 90b746efe2d54317b04ae5164cef1a3b: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260812 09:48:41.650446 9198 ts_tablet_manager.cc:1397] T 9e454f9dfb96467399fa4eb28c8ade3a P 6440e49253344afebefac34f6e39e878: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260812 09:48:41.653718 9201 raft_consensus.cc:348] T 9e454f9dfb96467399fa4eb28c8ade3a P 90b746efe2d54317b04ae5164cef1a3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b746efe2d54317b04ae5164cef1a3b" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 43737 } } peers { permanent_uuid: "6440e49253344afebefac34f6e39e878" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 40441 } } peers { permanent_uuid: "f9e2dce914b843b2bfc0de5f215a45ff" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 34429 } }
I20260812 09:48:41.653719 9198 raft_consensus.cc:348] T 9e454f9dfb96467399fa4eb28c8ade3a P 6440e49253344afebefac34f6e39e878 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b746efe2d54317b04ae5164cef1a3b" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 43737 } } peers { permanent_uuid: "6440e49253344afebefac34f6e39e878" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 40441 } } peers { permanent_uuid: "f9e2dce914b843b2bfc0de5f215a45ff" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 34429 } }
I20260812 09:48:41.653941 9198 raft_consensus.cc:374] T 9e454f9dfb96467399fa4eb28c8ade3a P 6440e49253344afebefac34f6e39e878 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:48:41.653970 9201 raft_consensus.cc:374] T 9e454f9dfb96467399fa4eb28c8ade3a P 90b746efe2d54317b04ae5164cef1a3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:48:41.654008 9198 raft_consensus.cc:729] T 9e454f9dfb96467399fa4eb28c8ade3a P 6440e49253344afebefac34f6e39e878 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6440e49253344afebefac34f6e39e878, State: Initialized, Role: FOLLOWER
I20260812 09:48:41.654055 9201 raft_consensus.cc:729] T 9e454f9dfb96467399fa4eb28c8ade3a P 90b746efe2d54317b04ae5164cef1a3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90b746efe2d54317b04ae5164cef1a3b, State: Initialized, Role: FOLLOWER
I20260812 09:48:41.654531 9198 consensus_queue.cc:260] T 9e454f9dfb96467399fa4eb28c8ade3a P 6440e49253344afebefac34f6e39e878 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b746efe2d54317b04ae5164cef1a3b" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 43737 } } peers { permanent_uuid: "6440e49253344afebefac34f6e39e878" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 40441 } } peers { permanent_uuid: "f9e2dce914b843b2bfc0de5f215a45ff" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 34429 } }
I20260812 09:48:41.654511 9201 consensus_queue.cc:260] T 9e454f9dfb96467399fa4eb28c8ade3a P 90b746efe2d54317b04ae5164cef1a3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b746efe2d54317b04ae5164cef1a3b" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 43737 } } peers { permanent_uuid: "6440e49253344afebefac34f6e39e878" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 40441 } } peers { permanent_uuid: "f9e2dce914b843b2bfc0de5f215a45ff" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 34429 } }
I20260812 09:48:41.655251 9176 heartbeater.cc:503] Master 127.13.99.62:37085 was elected leader, sending a full tablet report...
I20260812 09:48:41.655476 8937 heartbeater.cc:503] Master 127.13.99.62:37085 was elected leader, sending a full tablet report...
I20260812 09:48:41.655488 9198 ts_tablet_manager.cc:1428] T 9e454f9dfb96467399fa4eb28c8ade3a P 6440e49253344afebefac34f6e39e878: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260812 09:48:41.655510 9201 ts_tablet_manager.cc:1428] T 9e454f9dfb96467399fa4eb28c8ade3a P 90b746efe2d54317b04ae5164cef1a3b: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260812 09:48:41.678081 9208 raft_consensus.cc:482] T 9e454f9dfb96467399fa4eb28c8ade3a P 90b746efe2d54317b04ae5164cef1a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 09:48:41.678349 9208 raft_consensus.cc:504] T 9e454f9dfb96467399fa4eb28c8ade3a P 90b746efe2d54317b04ae5164cef1a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b746efe2d54317b04ae5164cef1a3b" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 43737 } } peers { permanent_uuid: "6440e49253344afebefac34f6e39e878" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 40441 } } peers { permanent_uuid: "f9e2dce914b843b2bfc0de5f215a45ff" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 34429 } }
I20260812 09:48:41.679591 9208 leader_election.cc:288] T 9e454f9dfb96467399fa4eb28c8ade3a P 90b746efe2d54317b04ae5164cef1a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6440e49253344afebefac34f6e39e878 (127.13.99.3:40441), f9e2dce914b843b2bfc0de5f215a45ff (127.13.99.1:34429)
I20260812 09:48:41.683856 8945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:41.679848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.3:40441 (local address 127.13.99.2:34251)
0812 09:48:41.679979 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0812 09:48:41.679997 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 09:48:41.680091 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:48:41.680370 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:48:41.680374 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:48:41.680397 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:48:41.680576 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:48:41.680583 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:48:41.681596 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:48:41.681599 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:48:41.682750 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:48:41.682761 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:48:41.683532 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:48:41.683535 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:48:41.683616 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:48:41.683637 (+ 21us) client_negotiation.cc:769] Sending connection context
0812 09:48:41.683699 (+ 62us) client_negotiation.cc:240] Negotiation successful
0812 09:48:41.683755 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260812 09:48:41.683969 9195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:41.679953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.2:34251 (local address 127.13.99.3:40441)
0812 09:48:41.680091 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0812 09:48:41.680096 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:48:41.680113 (+ 17us) server_negotiation.cc:407] Connection header received
0812 09:48:41.680178 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:48:41.680184 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:48:41.680243 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:48:41.680330 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:48:41.680696 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:48:41.681475 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:48:41.682883 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:48:41.683410 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:48:41.683520 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:48:41.683585 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0812 09:48:41.683744 (+ 159us) server_negotiation.cc:299] Negotiation successful
0812 09:48:41.683820 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260812 09:48:41.684592 9212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:41.679848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.1:34429 (local address 127.13.99.2:51219)
0812 09:48:41.680425 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0812 09:48:41.680440 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 09:48:41.680566 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:48:41.680896 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:48:41.680898 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:48:41.680914 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:48:41.681010 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:48:41.681014 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:48:41.681953 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:48:41.681957 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:48:41.683037 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:48:41.683045 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:48:41.684015 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:48:41.684020 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:48:41.684201 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:48:41.684223 (+ 22us) client_negotiation.cc:769] Sending connection context
0812 09:48:41.684349 (+ 126us) client_negotiation.cc:240] Negotiation successful
0812 09:48:41.684408 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"mutex_wait_us":165,"thread_start_us":151,"threads_started":1}
I20260812 09:48:41.684629 8722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:41.679960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.2:51219 (local address 127.13.99.1:34429)
0812 09:48:41.680102 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0812 09:48:41.680107 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:48:41.680544 (+ 437us) server_negotiation.cc:407] Connection header received
0812 09:48:41.680697 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:48:41.680700 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:48:41.680770 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:48:41.680876 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:48:41.681132 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:48:41.681809 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:48:41.683156 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:48:41.683836 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:48:41.683939 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:48:41.684114 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0812 09:48:41.684382 (+ 268us) server_negotiation.cc:299] Negotiation successful
0812 09:48:41.684501 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260812 09:48:41.684803 9097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e454f9dfb96467399fa4eb28c8ade3a" candidate_uuid: "90b746efe2d54317b04ae5164cef1a3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6440e49253344afebefac34f6e39e878" is_pre_election: true
I20260812 09:48:41.685137 9097 raft_consensus.cc:2454] T 9e454f9dfb96467399fa4eb28c8ade3a P 6440e49253344afebefac34f6e39e878 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90b746efe2d54317b04ae5164cef1a3b in term 0.
I20260812 09:48:41.685374 8611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e454f9dfb96467399fa4eb28c8ade3a" candidate_uuid: "90b746efe2d54317b04ae5164cef1a3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9e2dce914b843b2bfc0de5f215a45ff" is_pre_election: true
I20260812 09:48:41.685678 8611 raft_consensus.cc:2454] T 9e454f9dfb96467399fa4eb28c8ade3a P f9e2dce914b843b2bfc0de5f215a45ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90b746efe2d54317b04ae5164cef1a3b in term 0.
I20260812 09:48:41.685683 8750 leader_election.cc:302] T 9e454f9dfb96467399fa4eb28c8ade3a P 90b746efe2d54317b04ae5164cef1a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6440e49253344afebefac34f6e39e878, 90b746efe2d54317b04ae5164cef1a3b; no voters:
I20260812 09:48:41.685971 9208 raft_consensus.cc:2781] T 9e454f9dfb96467399fa4eb28c8ade3a P 90b746efe2d54317b04ae5164cef1a3b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 09:48:41.686072 9208 raft_consensus.cc:482] T 9e454f9dfb96467399fa4eb28c8ade3a P 90b746efe2d54317b04ae5164cef1a3b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 09:48:41.686136 9208 raft_consensus.cc:3037] T 9e454f9dfb96467399fa4eb28c8ade3a P 90b746efe2d54317b04ae5164cef1a3b [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:48:41.687211 9208 raft_consensus.cc:504] T 9e454f9dfb96467399fa4eb28c8ade3a P 90b746efe2d54317b04ae5164cef1a3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b746efe2d54317b04ae5164cef1a3b" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 43737 } } peers { permanent_uuid: "6440e49253344afebefac34f6e39e878" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 40441 } } peers { permanent_uuid: "f9e2dce914b843b2bfc0de5f215a45ff" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 34429 } }
I20260812 09:48:41.687595 9208 leader_election.cc:288] T 9e454f9dfb96467399fa4eb28c8ade3a P 90b746efe2d54317b04ae5164cef1a3b [CANDIDATE]: Term 1 election: Requested vote from peers 6440e49253344afebefac34f6e39e878 (127.13.99.3:40441), f9e2dce914b843b2bfc0de5f215a45ff (127.13.99.1:34429)
I20260812 09:48:41.687929 9097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e454f9dfb96467399fa4eb28c8ade3a" candidate_uuid: "90b746efe2d54317b04ae5164cef1a3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6440e49253344afebefac34f6e39e878"
I20260812 09:48:41.688076 9097 raft_consensus.cc:3037] T 9e454f9dfb96467399fa4eb28c8ade3a P 6440e49253344afebefac34f6e39e878 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:48:41.688088 8611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e454f9dfb96467399fa4eb28c8ade3a" candidate_uuid: "90b746efe2d54317b04ae5164cef1a3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9e2dce914b843b2bfc0de5f215a45ff"
I20260812 09:48:41.688238 8611 raft_consensus.cc:3037] T 9e454f9dfb96467399fa4eb28c8ade3a P f9e2dce914b843b2bfc0de5f215a45ff [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:48:41.689231 9097 raft_consensus.cc:2454] T 9e454f9dfb96467399fa4eb28c8ade3a P 6440e49253344afebefac34f6e39e878 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90b746efe2d54317b04ae5164cef1a3b in term 1.
I20260812 09:48:41.689436 8611 raft_consensus.cc:2454] T 9e454f9dfb96467399fa4eb28c8ade3a P f9e2dce914b843b2bfc0de5f215a45ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90b746efe2d54317b04ae5164cef1a3b in term 1.
I20260812 09:48:41.689581 8750 leader_election.cc:302] T 9e454f9dfb96467399fa4eb28c8ade3a P 90b746efe2d54317b04ae5164cef1a3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6440e49253344afebefac34f6e39e878, 90b746efe2d54317b04ae5164cef1a3b; no voters:
I20260812 09:48:41.689808 9208 raft_consensus.cc:2781] T 9e454f9dfb96467399fa4eb28c8ade3a P 90b746efe2d54317b04ae5164cef1a3b [term 1 FOLLOWER]: Leader election won for term 1
I20260812 09:48:41.690038 9208 raft_consensus.cc:686] T 9e454f9dfb96467399fa4eb28c8ade3a P 90b746efe2d54317b04ae5164cef1a3b [term 1 LEADER]: Becoming Leader. State: Replica: 90b746efe2d54317b04ae5164cef1a3b, State: Running, Role: LEADER
I20260812 09:48:41.690390 9208 consensus_queue.cc:237] T 9e454f9dfb96467399fa4eb28c8ade3a P 90b746efe2d54317b04ae5164cef1a3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b746efe2d54317b04ae5164cef1a3b" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 43737 } } peers { permanent_uuid: "6440e49253344afebefac34f6e39e878" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 40441 } } peers { permanent_uuid: "f9e2dce914b843b2bfc0de5f215a45ff" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 34429 } }
I20260812 09:48:41.693738 8352 catalog_manager.cc:5495] T 9e454f9dfb96467399fa4eb28c8ade3a P 90b746efe2d54317b04ae5164cef1a3b reported cstate change: term changed from 0 to 1, leader changed from <none> to 90b746efe2d54317b04ae5164cef1a3b (127.13.99.2). New cstate: current_term: 1 leader_uuid: "90b746efe2d54317b04ae5164cef1a3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b746efe2d54317b04ae5164cef1a3b" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 43737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6440e49253344afebefac34f6e39e878" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 40441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9e2dce914b843b2bfc0de5f215a45ff" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 34429 } health_report { overall_health: UNKNOWN } } }
W20260812 09:48:41.695309 8701 tablet.cc:2369] T 9e454f9dfb96467399fa4eb28c8ade3a P f9e2dce914b843b2bfc0de5f215a45ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 09:48:41.715634 9192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:41.712043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56412 (local address 127.13.99.2:43737)
0812 09:48:41.712258 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0812 09:48:41.712261 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 09:48:41.712275 (+ 14us) server_negotiation.cc:407] Connection header received
0812 09:48:41.712351 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:48:41.712353 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:48:41.712409 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:48:41.712543 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 09:48:41.712823 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:48:41.713334 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:48:41.714207 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:48:41.714441 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:48:41.714532 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:48:41.714823 (+ 291us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 09:48:41.715269 (+ 446us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 09:48:41.715363 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0812 09:48:41.715471 (+ 108us) server_negotiation.cc:299] Negotiation successful
0812 09:48:41.715523 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
I20260812 09:48:41.721987 9097 raft_consensus.cc:1261] T 9e454f9dfb96467399fa4eb28c8ade3a P 6440e49253344afebefac34f6e39e878 [term 1 FOLLOWER]: Refusing update from remote peer 90b746efe2d54317b04ae5164cef1a3b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 09:48:41.721992 8611 raft_consensus.cc:1261] T 9e454f9dfb96467399fa4eb28c8ade3a P f9e2dce914b843b2bfc0de5f215a45ff [term 1 FOLLOWER]: Refusing update from remote peer 90b746efe2d54317b04ae5164cef1a3b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 09:48:41.722767 9208 consensus_queue.cc:1035] T 9e454f9dfb96467399fa4eb28c8ade3a P 90b746efe2d54317b04ae5164cef1a3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9e2dce914b843b2bfc0de5f215a45ff" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 34429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 09:48:41.723039 9213 consensus_queue.cc:1035] T 9e454f9dfb96467399fa4eb28c8ade3a P 90b746efe2d54317b04ae5164cef1a3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6440e49253344afebefac34f6e39e878" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 40441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 09:48:41.728838 9216 mvcc.cc:205] Tried to move back new op lower bound from 7317619186565111808 to 7317619186447155200. Current Snapshot: MvccSnapshot[applied={T|T < 7317619186565111808}]
I20260812 09:48:41.741844 8350 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50992:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 09:48:41.742049 8350 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50992:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 09:48:41.744535 8350 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6dbe64111bfb4d4a8730f47f3a1a4fb0: Sending DeleteTablet for 3 replicas of tablet 9e454f9dfb96467399fa4eb28c8ade3a
I20260812 09:48:41.745178 8825 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e454f9dfb96467399fa4eb28c8ade3a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 09:48:41 UTC) from {username='kudu', principal='kudu/127.13.99.62@KRBTEST.COM'} at 127.0.0.1:56406
I20260812 09:48:41.745677 9234 tablet_replica.cc:330] T 9e454f9dfb96467399fa4eb28c8ade3a P 90b746efe2d54317b04ae5164cef1a3b: stopping tablet replica
I20260812 09:48:41.745694 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 8436
I20260812 09:48:41.745682 9067 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e454f9dfb96467399fa4eb28c8ade3a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 09:48:41 UTC) from {username='kudu', principal='kudu/127.13.99.62@KRBTEST.COM'} at 127.0.0.1:60694
I20260812 09:48:41.746151 9234 raft_consensus.cc:2229] T 9e454f9dfb96467399fa4eb28c8ade3a P 90b746efe2d54317b04ae5164cef1a3b [term 1 LEADER]: Raft consensus shutting down.
I20260812 09:48:41.746522 9234 raft_consensus.cc:2258] T 9e454f9dfb96467399fa4eb28c8ade3a P 90b746efe2d54317b04ae5164cef1a3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 09:48:41.757548 9235 tablet_replica.cc:330] T 9e454f9dfb96467399fa4eb28c8ade3a P 6440e49253344afebefac34f6e39e878: stopping tablet replica
I20260812 09:48:41.758101 9235 raft_consensus.cc:2229] T 9e454f9dfb96467399fa4eb28c8ade3a P 6440e49253344afebefac34f6e39e878 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 09:48:41.758560 9235 raft_consensus.cc:2258] T 9e454f9dfb96467399fa4eb28c8ade3a P 6440e49253344afebefac34f6e39e878 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 09:48:41.761667 9235 ts_tablet_manager.cc:1905] T 9e454f9dfb96467399fa4eb28c8ade3a P 6440e49253344afebefac34f6e39e878: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 09:48:41.765125 9235 ts_tablet_manager.cc:1918] T 9e454f9dfb96467399fa4eb28c8ade3a P 6440e49253344afebefac34f6e39e878: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 09:48:41.765232 9235 log.cc:1196] T 9e454f9dfb96467399fa4eb28c8ade3a P 6440e49253344afebefac34f6e39e878: Deleting WAL directory at /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-2/wal/wals/9e454f9dfb96467399fa4eb28c8ade3a
I20260812 09:48:41.765579 9235 ts_tablet_manager.cc:1939] T 9e454f9dfb96467399fa4eb28c8ade3a P 6440e49253344afebefac34f6e39e878: Deleting consensus metadata
I20260812 09:48:41.767313 8321 catalog_manager.cc:4841] TS 6440e49253344afebefac34f6e39e878 (127.13.99.3:40441): tablet 9e454f9dfb96467399fa4eb28c8ade3a (table test-table [id=3064472d0b14441a85372d4b7c4dc862]) successfully deleted
I20260812 09:48:41.769019 9234 ts_tablet_manager.cc:1905] T 9e454f9dfb96467399fa4eb28c8ade3a P 90b746efe2d54317b04ae5164cef1a3b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 09:48:41.772054 9234 ts_tablet_manager.cc:1918] T 9e454f9dfb96467399fa4eb28c8ade3a P 90b746efe2d54317b04ae5164cef1a3b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 09:48:41.772169 9234 log.cc:1196] T 9e454f9dfb96467399fa4eb28c8ade3a P 90b746efe2d54317b04ae5164cef1a3b: Deleting WAL directory at /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-1/wal/wals/9e454f9dfb96467399fa4eb28c8ade3a
I20260812 09:48:41.772558 9234 ts_tablet_manager.cc:1939] T 9e454f9dfb96467399fa4eb28c8ade3a P 90b746efe2d54317b04ae5164cef1a3b: Deleting consensus metadata
I20260812 09:48:41.773885 8319 catalog_manager.cc:4841] TS 90b746efe2d54317b04ae5164cef1a3b (127.13.99.2:43737): tablet 9e454f9dfb96467399fa4eb28c8ade3a (table test-table [id=3064472d0b14441a85372d4b7c4dc862]) successfully deleted
W20260812 09:48:41.774457 8322 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.99.1:34429 (error 108)
I20260812 09:48:41.774680 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 8708
I20260812 09:48:41.777398 9193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:48:41.775401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.1:34429 (local address 127.0.0.1:56534)
0812 09:48:41.777224 (+ 1823us) negotiation.cc:107] Waiting for socket to connect
0812 09:48:41.777305 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.1:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1721}
W20260812 09:48:41.779156 8322 catalog_manager.cc:4584] TS f9e2dce914b843b2bfc0de5f215a45ff (127.13.99.1:34429): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9e454f9dfb96467399fa4eb28c8ade3a: Network error: Client connection negotiation failed: client connection to 127.13.99.1:34429: connect: Connection refused (error 111)
I20260812 09:48:41.788044 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 8949
I20260812 09:48:41.806593 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 8283
I20260812 09:49:01.822897 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.62:37085
--webserver_interface=127.13.99.62
--webserver_port=34893
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab
--principal=kudu/127.13.99.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:39987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.99.62:37085
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:49:01.939050 9237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:49:01.939342 9237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:49:01.939383 9237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:49:01.943501 9237 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260812 09:49:01.943562 9237 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260812 09:49:01.943579 9237 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260812 09:49:01.943593 9237 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 09:49:01.943611 9237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:49:01.943670 9237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:49:01.943699 9237 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 09:49:01.943732 9237 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 09:49:01.949147 9237 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:39987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.13.99.62:37085
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab
--principal=kudu/127.13.99.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.62:37085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.99.62
--webserver_port=34893
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9237
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:49:01.950173 9237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:49:01.951222 9237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:49:01.960583 9242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:49:01.960590 9243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:49:01.960717 9237 server_base.cc:1034] running on GCE node
W20260812 09:49:01.960855 9245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:49:01.961606 9237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:49:01.962782 9237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:49:01.963975 9237 hybrid_clock.cc:648] HybridClock initialized: now 1786528141963962 us; error 27 us; skew 500 ppm
Aug 12 09:49:01 dist-test-slave-jq4t krb5kdc[7284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528141, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:49:01.970990 9237 init.cc:377] Logged in from keytab as kudu/127.13.99.62@KRBTEST.COM (short username kudu)
I20260812 09:49:01.971915 9237 webserver.cc:460] Webserver started at http://127.13.99.62:34893/ using document root <none> and password file <none>
I20260812 09:49:01.972559 9237 fs_manager.cc:359] Metadata directory not provided
I20260812 09:49:01.972656 9237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:49:01.976516 9237 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260812 09:49:01.978724 9258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:49:01.979876 9237 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 09:49:01.980015 9237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/master-0/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/master-0/wal
uuid: "6dbe64111bfb4d4a8730f47f3a1a4fb0"
format_stamp: "Formatted at 2026-08-12 09:48:19 on dist-test-slave-jq4t"
server_key: "e48ace4008e2743289311f3907fe64d7"
server_key_iv: "935ea577c777a918b942eb11a9b78c31"
server_key_version: "encryptionkey@0"
I20260812 09:49:01.980619 9237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:49:01.990660 9237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:49:01.993862 9237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:49:01.994077 9237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:49:02.001293 9237 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.62:37085
I20260812 09:49:02.001580 9351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.62:37085 every 8 connection(s)
I20260812 09:49:02.002420 9237 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
I20260812 09:49:02.006858 9352 sys_catalog.cc:263] Verifying existing consensus state
I20260812 09:49:02.008577 9352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6dbe64111bfb4d4a8730f47f3a1a4fb0: Bootstrap starting.
I20260812 09:49:02.010718 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 9237
I20260812 09:49:02.011256 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.99.1:34429
--local_ip_for_outbound_sockets=127.13.99.1
--tserver_master_addrs=127.13.99.62:37085
--webserver_port=36855
--webserver_interface=127.13.99.1
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.1.keytab
--principal=kudu/127.13.99.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:39987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 09:49:02.028270 9352 log.cc:824] T 00000000000000000000000000000000 P 6dbe64111bfb4d4a8730f47f3a1a4fb0: Log is configured to *not* fsync() on all Append() calls
I20260812 09:49:02.039491 9352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6dbe64111bfb4d4a8730f47f3a1a4fb0: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 09:49:02.039867 9352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6dbe64111bfb4d4a8730f47f3a1a4fb0: Bootstrap complete.
I20260812 09:49:02.044008 9352 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6dbe64111bfb4d4a8730f47f3a1a4fb0 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dbe64111bfb4d4a8730f47f3a1a4fb0" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 37085 } }
I20260812 09:49:02.044382 9352 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6dbe64111bfb4d4a8730f47f3a1a4fb0 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dbe64111bfb4d4a8730f47f3a1a4fb0, State: Initialized, Role: FOLLOWER
I20260812 09:49:02.044957 9352 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6dbe64111bfb4d4a8730f47f3a1a4fb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dbe64111bfb4d4a8730f47f3a1a4fb0" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 37085 } }
I20260812 09:49:02.045080 9352 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6dbe64111bfb4d4a8730f47f3a1a4fb0 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 09:49:02.045164 9352 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6dbe64111bfb4d4a8730f47f3a1a4fb0 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 09:49:02.045275 9352 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6dbe64111bfb4d4a8730f47f3a1a4fb0 [term 2 FOLLOWER]: Advancing to term 3
I20260812 09:49:02.047119 9352 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6dbe64111bfb4d4a8730f47f3a1a4fb0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dbe64111bfb4d4a8730f47f3a1a4fb0" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 37085 } }
I20260812 09:49:02.047500 9352 leader_election.cc:302] T 00000000000000000000000000000000 P 6dbe64111bfb4d4a8730f47f3a1a4fb0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6dbe64111bfb4d4a8730f47f3a1a4fb0; no voters:
I20260812 09:49:02.047806 9352 leader_election.cc:288] T 00000000000000000000000000000000 P 6dbe64111bfb4d4a8730f47f3a1a4fb0 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260812 09:49:02.047957 9359 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6dbe64111bfb4d4a8730f47f3a1a4fb0 [term 3 FOLLOWER]: Leader election won for term 3
I20260812 09:49:02.048208 9359 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6dbe64111bfb4d4a8730f47f3a1a4fb0 [term 3 LEADER]: Becoming Leader. State: Replica: 6dbe64111bfb4d4a8730f47f3a1a4fb0, State: Running, Role: LEADER
I20260812 09:49:02.048642 9359 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6dbe64111bfb4d4a8730f47f3a1a4fb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dbe64111bfb4d4a8730f47f3a1a4fb0" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 37085 } }
I20260812 09:49:02.049036 9352 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6dbe64111bfb4d4a8730f47f3a1a4fb0 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 09:49:02.050073 9361 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dbe64111bfb4d4a8730f47f3a1a4fb0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6dbe64111bfb4d4a8730f47f3a1a4fb0. Latest consensus state: current_term: 3 leader_uuid: "6dbe64111bfb4d4a8730f47f3a1a4fb0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dbe64111bfb4d4a8730f47f3a1a4fb0" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 37085 } } }
I20260812 09:49:02.050201 9361 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dbe64111bfb4d4a8730f47f3a1a4fb0 [sys.catalog]: This master's current role is: LEADER
I20260812 09:49:02.050922 9360 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dbe64111bfb4d4a8730f47f3a1a4fb0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "6dbe64111bfb4d4a8730f47f3a1a4fb0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dbe64111bfb4d4a8730f47f3a1a4fb0" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 37085 } } }
I20260812 09:49:02.051012 9360 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dbe64111bfb4d4a8730f47f3a1a4fb0 [sys.catalog]: This master's current role is: LEADER
I20260812 09:49:02.051506 9375 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 09:49:02.057126 9375 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 09:49:02.057760 9375 catalog_manager.cc:1259] Loaded cluster ID: 68a068ddd80a4047bf4e04d78077accc
I20260812 09:49:02.057898 9375 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 09:49:02.061058 9375 catalog_manager.cc:1504] Loading token signing keys...
I20260812 09:49:02.061988 9375 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6dbe64111bfb4d4a8730f47f3a1a4fb0: Loaded TSK: 0
I20260812 09:49:02.062086 9375 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6dbe64111bfb4d4a8730f47f3a1a4fb0: Loaded TSK: 1
F20260812 09:49:02.062872 9375 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f3fd71639ad google::LogMessage::Fail() at ??:0
@ 0x7f3fd716688a google::LogMessage::SendToLog() at ??:0
@ 0x7f3fd71634de google::LogMessage::Flush() at ??:0
@ 0x7f3fd7164a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f3fd9e2718d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f3fd9e1ff0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f3fe1ac87bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f3fe1aa299c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f3fe1aa3178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f3fe1ae4df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f3fe1b0382b std::function<>::operator()() at ??:0
@ 0x7f3fe1aa2b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f3fe1aa3f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f3fe1aa0579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f3fe1ae4640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f3fd93b53d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f3fd93b5c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f3fd93b73f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f3fd93a6512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f3fda9ea6db start_thread at ??:0
@ 0x7f3fd5c6161f clone at ??:0
W20260812 09:49:02.166757 9357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:49:02.167001 9357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:49:02.167049 9357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:49:02.171079 9357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:49:02.171129 9357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:49:02.171212 9357 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.99.1
I20260812 09:49:02.176381 9357 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:39987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.1.keytab
--principal=kudu/127.13.99.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.99.1:34429
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.99.1
--webserver_port=36855
--tserver_master_addrs=127.13.99.62:37085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9357
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.99.1
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:49:02.177469 9357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:49:02.178499 9357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:49:02.188099 9439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:49:02.188151 9435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:49:02.188382 9357 server_base.cc:1034] running on GCE node
W20260812 09:49:02.188396 9441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:49:02.188836 9357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:49:02.189509 9357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:49:02.190696 9357 hybrid_clock.cc:648] HybridClock initialized: now 1786528142190664 us; error 50 us; skew 500 ppm
Aug 12 09:49:02 dist-test-slave-jq4t krb5kdc[7284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528142, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:49:02.193665 9357 init.cc:377] Logged in from keytab as kudu/127.13.99.1@KRBTEST.COM (short username kudu)
I20260812 09:49:02.194568 9357 webserver.cc:460] Webserver started at http://127.13.99.1:36855/ using document root <none> and password file <none>
I20260812 09:49:02.195142 9357 fs_manager.cc:359] Metadata directory not provided
I20260812 09:49:02.195206 9357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:49:02.199138 9357 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.001s
I20260812 09:49:02.201421 9449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:49:02.202700 9357 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260812 09:49:02.202836 9357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-0/wal
uuid: "f9e2dce914b843b2bfc0de5f215a45ff"
format_stamp: "Formatted at 2026-08-12 09:48:19 on dist-test-slave-jq4t"
server_key: "493d29e9ebcdf3ef9dadeb75241b9724"
server_key_iv: "322088c3e4178e5fd7efe60477020e3b"
server_key_version: "encryptionkey@0"
I20260812 09:49:02.203382 9357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:49:02.225406 9357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:49:02.228895 9357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:49:02.229107 9357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:49:02.229763 9357 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 09:49:02.230963 9462 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260812 09:49:02.236701 9357 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 09:49:02.236773 9357 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260812 09:49:02.236845 9357 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260812 09:49:02.238547 9357 ts_tablet_manager.cc:610] Registered 1 tablets
I20260812 09:49:02.238618 9357 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260812 09:49:02.238926 9462 tablet_bootstrap.cc:492] T 9e454f9dfb96467399fa4eb28c8ade3a P f9e2dce914b843b2bfc0de5f215a45ff: Bootstrap starting.
I20260812 09:49:02.249861 9357 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.1:34429
I20260812 09:49:02.250191 9648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.1:34429 every 8 connection(s)
I20260812 09:49:02.250968 9357 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-0/data/info.pb
I20260812 09:49:02.259431 9462 log.cc:824] T 9e454f9dfb96467399fa4eb28c8ade3a P f9e2dce914b843b2bfc0de5f215a45ff: Log is configured to *not* fsync() on all Append() calls
I20260812 09:49:02.260371 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 9357
I20260812 09:49:02.260844 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.99.2:43737
--local_ip_for_outbound_sockets=127.13.99.2
--tserver_master_addrs=127.13.99.62:37085
--webserver_port=37537
--webserver_interface=127.13.99.2
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.2.keytab
--principal=kudu/127.13.99.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:39987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 09:49:02.268721 9462 tablet_bootstrap.cc:492] T 9e454f9dfb96467399fa4eb28c8ade3a P f9e2dce914b843b2bfc0de5f215a45ff: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 09:49:02.269170 9462 tablet_bootstrap.cc:492] T 9e454f9dfb96467399fa4eb28c8ade3a P f9e2dce914b843b2bfc0de5f215a45ff: Bootstrap complete.
I20260812 09:49:02.270561 9462 ts_tablet_manager.cc:1397] T 9e454f9dfb96467399fa4eb28c8ade3a P f9e2dce914b843b2bfc0de5f215a45ff: Time spent bootstrapping tablet: real 0.032s user 0.021s sys 0.008s
I20260812 09:49:02.273705 9462 raft_consensus.cc:348] T 9e454f9dfb96467399fa4eb28c8ade3a P f9e2dce914b843b2bfc0de5f215a45ff [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b746efe2d54317b04ae5164cef1a3b" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 43737 } } peers { permanent_uuid: "6440e49253344afebefac34f6e39e878" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 40441 } } peers { permanent_uuid: "f9e2dce914b843b2bfc0de5f215a45ff" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 34429 } }
I20260812 09:49:02.274251 9462 raft_consensus.cc:729] T 9e454f9dfb96467399fa4eb28c8ade3a P f9e2dce914b843b2bfc0de5f215a45ff [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9e2dce914b843b2bfc0de5f215a45ff, State: Initialized, Role: FOLLOWER
I20260812 09:49:02.274794 9462 consensus_queue.cc:260] T 9e454f9dfb96467399fa4eb28c8ade3a P f9e2dce914b843b2bfc0de5f215a45ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b746efe2d54317b04ae5164cef1a3b" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 43737 } } peers { permanent_uuid: "6440e49253344afebefac34f6e39e878" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 40441 } } peers { permanent_uuid: "f9e2dce914b843b2bfc0de5f215a45ff" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 34429 } }
I20260812 09:49:02.275943 9462 ts_tablet_manager.cc:1428] T 9e454f9dfb96467399fa4eb28c8ade3a P f9e2dce914b843b2bfc0de5f215a45ff: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
W20260812 09:49:02.304141 9654 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 09:49:02.253155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:37085 (local address 127.13.99.1:51923)
0812 09:49:02.253745 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:02.253811 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0812 09:49:02.254881 (+ 1070us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:49:02.303164 (+ 48283us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:37085: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":149,"threads_started":1}
W20260812 09:49:02.304752 9649 heartbeater.cc:644] Failed to heartbeat to 127.13.99.62:37085 (0 consecutive failures): Network error: Failed to ping master at 127.13.99.62:37085: Client connection negotiation failed: client connection to 127.13.99.62:37085: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 09:49:02.305559 9654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:02.305294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:37085 (local address 127.13.99.1:43287)
0812 09:49:02.305426 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:02.305493 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:37085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260812 09:49:02.306249 9654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:02.306018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:37085 (local address 127.13.99.1:42887)
0812 09:49:02.306146 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:02.306201 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:37085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260812 09:49:02.306449 9649 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 09:49:02.379822 9664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:49:02.380110 9664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:49:02.380191 9664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:49:02.384210 9664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:49:02.384290 9664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:49:02.384382 9664 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.99.2
I20260812 09:49:02.389590 9664 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:39987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.2.keytab
--principal=kudu/127.13.99.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.99.2:43737
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.99.2
--webserver_port=37537
--tserver_master_addrs=127.13.99.62:37085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9664
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.99.2
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:49:02.390633 9664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:49:02.391660 9664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:49:02.401434 9686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:49:02.401454 9683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:49:02.401474 9682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:49:02.401613 9664 server_base.cc:1034] running on GCE node
I20260812 09:49:02.402128 9664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:49:02.402789 9664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:49:02.403975 9664 hybrid_clock.cc:648] HybridClock initialized: now 1786528142403952 us; error 40 us; skew 500 ppm
Aug 12 09:49:02 dist-test-slave-jq4t krb5kdc[7284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528142, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:49:02.406826 9664 init.cc:377] Logged in from keytab as kudu/127.13.99.2@KRBTEST.COM (short username kudu)
I20260812 09:49:02.407775 9664 webserver.cc:460] Webserver started at http://127.13.99.2:37537/ using document root <none> and password file <none>
I20260812 09:49:02.408355 9664 fs_manager.cc:359] Metadata directory not provided
I20260812 09:49:02.408447 9664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:49:02.412441 9664 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260812 09:49:02.414628 9700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:49:02.415859 9664 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 09:49:02.415994 9664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-1/wal
uuid: "90b746efe2d54317b04ae5164cef1a3b"
format_stamp: "Formatted at 2026-08-12 09:48:20 on dist-test-slave-jq4t"
server_key: "2304f1d32d8d1d9ae92fb41ef8950a57"
server_key_iv: "98ab55e79a37ce873231623ae8b1522c"
server_key_version: "encryptionkey@0"
I20260812 09:49:02.416610 9664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:49:02.450911 9664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:49:02.454507 9664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:49:02.454717 9664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:49:02.455356 9664 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 09:49:02.456249 9664 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:49:02.456334 9664 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 09:49:02.456424 9664 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:49:02.456471 9664 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 09:49:02.466168 9664 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.2:43737
I20260812 09:49:02.466202 9876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.2:43737 every 8 connection(s)
I20260812 09:49:02.467196 9664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-1/data/info.pb
I20260812 09:49:02.469590 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 9664
I20260812 09:49:02.470094 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.99.3:40441
--local_ip_for_outbound_sockets=127.13.99.3
--tserver_master_addrs=127.13.99.62:37085
--webserver_port=34537
--webserver_interface=127.13.99.3
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.3.keytab
--principal=kudu/127.13.99.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:39987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 09:49:02.471378 9880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:02.469411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:37085 (local address 127.13.99.2:52563)
0812 09:49:02.469988 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:02.470101 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:37085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":156,"threads_started":1}
W20260812 09:49:02.471907 9877 heartbeater.cc:644] Failed to heartbeat to 127.13.99.62:37085 (0 consecutive failures): Network error: Failed to ping master at 127.13.99.62:37085: Client connection negotiation failed: client connection to 127.13.99.62:37085: connect: Connection refused (error 111)
I20260812 09:49:02.472612 9880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:02.472307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:37085 (local address 127.13.99.2:58817)
0812 09:49:02.472441 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:02.472535 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:37085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260812 09:49:02.473278 9880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:02.473016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:37085 (local address 127.13.99.2:33887)
0812 09:49:02.473142 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:02.473208 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:37085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260812 09:49:02.473459 9877 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 09:49:02.588114 9881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:49:02.588376 9881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:49:02.588446 9881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:49:02.592362 9881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:49:02.592440 9881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:49:02.592541 9881 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.99.3
I20260812 09:49:02.597731 9881 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:39987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.3.keytab
--principal=kudu/127.13.99.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.99.3:40441
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.99.3
--webserver_port=34537
--tserver_master_addrs=127.13.99.62:37085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9881
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.99.3
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:49:02.598706 9881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:49:02.599681 9881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:49:02.609678 9894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:49:02.609738 9889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:49:02.609702 9888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:49:02.609902 9881 server_base.cc:1034] running on GCE node
I20260812 09:49:02.610523 9881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:49:02.611228 9881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:49:02.612432 9881 hybrid_clock.cc:648] HybridClock initialized: now 1786528142612415 us; error 31 us; skew 500 ppm
Aug 12 09:49:02 dist-test-slave-jq4t krb5kdc[7284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528142, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:49:02.615329 9881 init.cc:377] Logged in from keytab as kudu/127.13.99.3@KRBTEST.COM (short username kudu)
I20260812 09:49:02.616247 9881 webserver.cc:460] Webserver started at http://127.13.99.3:34537/ using document root <none> and password file <none>
I20260812 09:49:02.616886 9881 fs_manager.cc:359] Metadata directory not provided
I20260812 09:49:02.616990 9881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:49:02.621049 9881 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 09:49:02.623137 9904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:49:02.624280 9881 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 09:49:02.624413 9881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-2/wal
uuid: "6440e49253344afebefac34f6e39e878"
format_stamp: "Formatted at 2026-08-12 09:48:20 on dist-test-slave-jq4t"
server_key: "523a68247cb62087c9374fbdf31e43b3"
server_key_iv: "c08b041a9e4a37ba82a2e3f18c15f5fb"
server_key_version: "encryptionkey@0"
I20260812 09:49:02.625022 9881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:49:02.665215 9881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:49:02.668833 9881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:49:02.669044 9881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:49:02.669749 9881 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 09:49:02.670657 9881 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:49:02.670737 9881 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 09:49:02.670815 9881 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:49:02.670854 9881 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 09:49:02.680226 9881 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.3:40441
I20260812 09:49:02.680258 10098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.3:40441 every 8 connection(s)
I20260812 09:49:02.681247 9881 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786527905902985-13708-0/minicluster-data/ts-2/data/info.pb
I20260812 09:49:02.685077 10101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:02.683378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:37085 (local address 127.13.99.3:40273)
0812 09:49:02.683849 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:02.684346 (+ 497us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:37085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":120,"threads_started":1}
W20260812 09:49:02.685570 10099 heartbeater.cc:644] Failed to heartbeat to 127.13.99.62:37085 (0 consecutive failures): Network error: Failed to ping master at 127.13.99.62:37085: Client connection negotiation failed: client connection to 127.13.99.62:37085: connect: Connection refused (error 111)
I20260812 09:49:02.686137 10101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:02.685939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:37085 (local address 127.13.99.3:34203)
0812 09:49:02.686037 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:02.686085 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:37085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260812 09:49:02.686750 10101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:02.686566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:37085 (local address 127.13.99.3:35795)
0812 09:49:02.686661 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:02.686703 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:37085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260812 09:49:02.686956 10099 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260812 09:49:02.688974 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 9881
I20260812 09:49:03.307554 10103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:03.307033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:37085 (local address 127.13.99.1:52273)
0812 09:49:03.307307 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:03.307434 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:37085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":77,"threads_started":1}
I20260812 09:49:03.474459 10104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:03.474012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:37085 (local address 127.13.99.2:41709)
0812 09:49:03.474289 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:03.474347 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:37085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":70,"threads_started":1}
I20260812 09:49:03.688004 10105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:03.687558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:37085 (local address 127.13.99.3:34421)
0812 09:49:03.687826 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:03.687887 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:37085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":92,"threads_started":1}
I20260812 09:49:03.884663 10106 raft_consensus.cc:482] T 9e454f9dfb96467399fa4eb28c8ade3a P f9e2dce914b843b2bfc0de5f215a45ff [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 09:49:03.884896 10106 raft_consensus.cc:504] T 9e454f9dfb96467399fa4eb28c8ade3a P f9e2dce914b843b2bfc0de5f215a45ff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b746efe2d54317b04ae5164cef1a3b" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 43737 } } peers { permanent_uuid: "6440e49253344afebefac34f6e39e878" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 40441 } } peers { permanent_uuid: "f9e2dce914b843b2bfc0de5f215a45ff" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 34429 } }
I20260812 09:49:03.886113 10106 leader_election.cc:288] T 9e454f9dfb96467399fa4eb28c8ade3a P f9e2dce914b843b2bfc0de5f215a45ff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 90b746efe2d54317b04ae5164cef1a3b (127.13.99.2:43737), 6440e49253344afebefac34f6e39e878 (127.13.99.3:40441)
Aug 12 09:49:03 dist-test-slave-jq4t krb5kdc[7284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528142, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.1@KRBTEST.COM for kudu/127.13.99.3@KRBTEST.COM
Aug 12 09:49:03 dist-test-slave-jq4t krb5kdc[7284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528142, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.1@KRBTEST.COM for kudu/127.13.99.2@KRBTEST.COM
I20260812 09:49:03.900432 10111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:03.886437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.3:40441 (local address 127.13.99.1:53611)
0812 09:49:03.886793 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:03.886808 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 09:49:03.886923 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:49:03.888758 (+ 1835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:49:03.888765 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:49:03.889353 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:49:03.889867 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:49:03.889882 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:49:03.891186 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:49:03.891189 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:49:03.892176 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:49:03.892188 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:49:03.892959 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:49:03.892963 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:49:03.893211 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:03.893877 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:49:03.893893 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:49:03.895452 (+ 1559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:49:03.897727 (+ 2275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:49:03.897733 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:49:03.897748 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:49:03.898010 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:49:03.898367 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:49:03.898371 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:49:03.898373 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:49:03.898557 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:49:03.899368 (+ 811us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:49:03.899375 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:49:03.899667 (+ 292us) client_negotiation.cc:769] Sending connection context
0812 09:49:03.899854 (+ 187us) client_negotiation.cc:240] Negotiation successful
0812 09:49:03.900220 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"mutex_wait_us":141,"spinlock_wait_cycles":27136,"thread_start_us":64,"threads_started":1}
I20260812 09:49:03.900441 10112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:03.886517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.1:45425 (local address 127.13.99.2:43737)
0812 09:49:03.887009 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:03.887038 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:03.887088 (+ 50us) server_negotiation.cc:407] Connection header received
0812 09:49:03.887952 (+ 864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:03.887979 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:03.888349 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:03.888705 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:49:03.890049 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:03.891048 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:03.892310 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:03.892833 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:03.893010 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:03.896024 (+ 3014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:49:03.896028 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:49:03.896060 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:49:03.896099 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:49:03.897930 (+ 1831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:49:03.898489 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:49:03.898494 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:49:03.898498 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:49:03.898606 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:49:03.898951 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:49:03.898956 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:49:03.898958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:49:03.899390 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:49:03.899514 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:03.899978 (+ 464us) server_negotiation.cc:299] Negotiation successful
0812 09:49:03.900254 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":130,"threads_started":1}
I20260812 09:49:03.900465 10113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:03.886636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.1:53611 (local address 127.13.99.3:40441)
0812 09:49:03.887076 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:03.887096 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:03.887129 (+ 33us) server_negotiation.cc:407] Connection header received
0812 09:49:03.887933 (+ 804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:03.887943 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:03.888349 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:03.888705 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:49:03.890056 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:03.891035 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:03.892303 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:03.892831 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:03.893022 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:03.895724 (+ 2702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:49:03.895729 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:49:03.895742 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:49:03.895769 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:49:03.897599 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:49:03.898136 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:49:03.898142 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:49:03.898146 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:49:03.898264 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:49:03.898758 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:49:03.898762 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:49:03.898763 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:49:03.899227 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:49:03.899330 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:03.899983 (+ 653us) server_negotiation.cc:299] Negotiation successful
0812 09:49:03.900276 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":104,"threads_started":1}
I20260812 09:49:03.900499 10107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:03.886292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.2:43737 (local address 127.13.99.1:45425)
0812 09:49:03.886696 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:03.886713 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 09:49:03.886831 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:49:03.888758 (+ 1927us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:49:03.888765 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:49:03.889353 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:49:03.889868 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:49:03.889900 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:49:03.891195 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:49:03.891198 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:49:03.892174 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:49:03.892186 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:49:03.892959 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:49:03.892963 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:49:03.893210 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:03.893933 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:49:03.893946 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:49:03.895839 (+ 1893us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:49:03.898060 (+ 2221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:49:03.898063 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:49:03.898065 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:49:03.898325 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:49:03.898728 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:49:03.898731 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:49:03.898733 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:49:03.898821 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:49:03.899506 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:49:03.899509 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:49:03.899665 (+ 156us) client_negotiation.cc:769] Sending connection context
0812 09:49:03.899854 (+ 189us) client_negotiation.cc:240] Negotiation successful
0812 09:49:03.900302 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"mutex_wait_us":398,"spinlock_wait_cycles":8064,"thread_start_us":108,"threads_started":1}
W20260812 09:49:03.903098 9452 leader_election.cc:341] T 9e454f9dfb96467399fa4eb28c8ade3a P f9e2dce914b843b2bfc0de5f215a45ff [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 90b746efe2d54317b04ae5164cef1a3b (127.13.99.2:43737): Not found: Tablet not found: 9e454f9dfb96467399fa4eb28c8ade3a
W20260812 09:49:03.903249 9456 leader_election.cc:341] T 9e454f9dfb96467399fa4eb28c8ade3a P f9e2dce914b843b2bfc0de5f215a45ff [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6440e49253344afebefac34f6e39e878 (127.13.99.3:40441): Not found: Tablet not found: 9e454f9dfb96467399fa4eb28c8ade3a
I20260812 09:49:03.903323 9456 leader_election.cc:302] T 9e454f9dfb96467399fa4eb28c8ade3a P f9e2dce914b843b2bfc0de5f215a45ff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9e2dce914b843b2bfc0de5f215a45ff; no voters: 6440e49253344afebefac34f6e39e878, 90b746efe2d54317b04ae5164cef1a3b
I20260812 09:49:03.903543 10106 raft_consensus.cc:2726] T 9e454f9dfb96467399fa4eb28c8ade3a P f9e2dce914b843b2bfc0de5f215a45ff [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 09:49:04.308646 10107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:04.308311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:37085 (local address 127.13.99.1:48757)
0812 09:49:04.308458 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:04.308555 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:37085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260812 09:49:04.475519 10114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:04.475136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:37085 (local address 127.13.99.2:47743)
0812 09:49:04.475383 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:04.475436 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:37085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":71,"threads_started":1}
I20260812 09:49:04.689239 10117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:04.688788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:37085 (local address 127.13.99.3:54679)
0812 09:49:04.689059 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:04.689140 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:37085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":75,"threads_started":1}
I20260812 09:49:05.309804 10118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:05.309384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:37085 (local address 127.13.99.1:60587)
0812 09:49:05.309649 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:05.309706 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:37085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":86,"threads_started":1}
I20260812 09:49:05.476673 10119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:05.476234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:37085 (local address 127.13.99.2:56997)
0812 09:49:05.476529 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:05.476585 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:37085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":72,"threads_started":1}
I20260812 09:49:05.541996 10122 raft_consensus.cc:482] T 9e454f9dfb96467399fa4eb28c8ade3a P f9e2dce914b843b2bfc0de5f215a45ff [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 09:49:05.542147 10122 raft_consensus.cc:504] T 9e454f9dfb96467399fa4eb28c8ade3a P f9e2dce914b843b2bfc0de5f215a45ff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b746efe2d54317b04ae5164cef1a3b" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 43737 } } peers { permanent_uuid: "6440e49253344afebefac34f6e39e878" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 40441 } } peers { permanent_uuid: "f9e2dce914b843b2bfc0de5f215a45ff" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 34429 } }
I20260812 09:49:05.542699 10122 leader_election.cc:288] T 9e454f9dfb96467399fa4eb28c8ade3a P f9e2dce914b843b2bfc0de5f215a45ff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 90b746efe2d54317b04ae5164cef1a3b (127.13.99.2:43737), 6440e49253344afebefac34f6e39e878 (127.13.99.3:40441)
W20260812 09:49:05.543617 9456 leader_election.cc:341] T 9e454f9dfb96467399fa4eb28c8ade3a P f9e2dce914b843b2bfc0de5f215a45ff [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6440e49253344afebefac34f6e39e878 (127.13.99.3:40441): Not found: Tablet not found: 9e454f9dfb96467399fa4eb28c8ade3a
W20260812 09:49:05.543708 9452 leader_election.cc:341] T 9e454f9dfb96467399fa4eb28c8ade3a P f9e2dce914b843b2bfc0de5f215a45ff [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 90b746efe2d54317b04ae5164cef1a3b (127.13.99.2:43737): Not found: Tablet not found: 9e454f9dfb96467399fa4eb28c8ade3a
I20260812 09:49:05.543761 9456 leader_election.cc:302] T 9e454f9dfb96467399fa4eb28c8ade3a P f9e2dce914b843b2bfc0de5f215a45ff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9e2dce914b843b2bfc0de5f215a45ff; no voters: 6440e49253344afebefac34f6e39e878, 90b746efe2d54317b04ae5164cef1a3b
I20260812 09:49:05.543941 10122 raft_consensus.cc:2726] T 9e454f9dfb96467399fa4eb28c8ade3a P f9e2dce914b843b2bfc0de5f215a45ff [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 09:49:05.690310 10123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:05.689932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:37085 (local address 127.13.99.3:46359)
0812 09:49:05.690160 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:05.690212 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:37085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":77,"threads_started":1}
I20260812 09:49:06.311088 10124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:06.310643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:37085 (local address 127.13.99.1:52007)
0812 09:49:06.310926 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:06.310986 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:37085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":77,"threads_started":1}
I20260812 09:49:06.477769 10125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:06.477357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:37085 (local address 127.13.99.2:60903)
0812 09:49:06.477630 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:06.477684 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:37085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":97,"threads_started":1}
I20260812 09:49:06.691490 10126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:06.691063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:37085 (local address 127.13.99.3:60157)
0812 09:49:06.691337 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:06.691395 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:37085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":79,"threads_started":1}
I20260812 09:49:07.312453 10127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:07.311968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:37085 (local address 127.13.99.1:52519)
0812 09:49:07.312290 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:07.312351 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:37085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":92,"threads_started":1}
I20260812 09:49:07.478940 10128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:07.478482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:37085 (local address 127.13.99.2:50033)
0812 09:49:07.478778 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:07.478850 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:37085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":100,"threads_started":1}
I20260812 09:49:07.692811 10129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:07.692371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:37085 (local address 127.13.99.3:34001)
0812 09:49:07.692657 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:07.692715 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:37085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":96,"threads_started":1}
I20260812 09:49:08.313647 10134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:08.313215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:37085 (local address 127.13.99.1:49683)
0812 09:49:08.313490 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:08.313546 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:37085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":85,"threads_started":1}
I20260812 09:49:08.314087 10135 raft_consensus.cc:482] T 9e454f9dfb96467399fa4eb28c8ade3a P f9e2dce914b843b2bfc0de5f215a45ff [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 09:49:08.314219 10135 raft_consensus.cc:504] T 9e454f9dfb96467399fa4eb28c8ade3a P f9e2dce914b843b2bfc0de5f215a45ff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b746efe2d54317b04ae5164cef1a3b" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 43737 } } peers { permanent_uuid: "6440e49253344afebefac34f6e39e878" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 40441 } } peers { permanent_uuid: "f9e2dce914b843b2bfc0de5f215a45ff" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 34429 } }
I20260812 09:49:08.314672 10135 leader_election.cc:288] T 9e454f9dfb96467399fa4eb28c8ade3a P f9e2dce914b843b2bfc0de5f215a45ff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 90b746efe2d54317b04ae5164cef1a3b (127.13.99.2:43737), 6440e49253344afebefac34f6e39e878 (127.13.99.3:40441)
W20260812 09:49:08.315506 9452 leader_election.cc:341] T 9e454f9dfb96467399fa4eb28c8ade3a P f9e2dce914b843b2bfc0de5f215a45ff [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 90b746efe2d54317b04ae5164cef1a3b (127.13.99.2:43737): Not found: Tablet not found: 9e454f9dfb96467399fa4eb28c8ade3a
W20260812 09:49:08.315685 9456 leader_election.cc:341] T 9e454f9dfb96467399fa4eb28c8ade3a P f9e2dce914b843b2bfc0de5f215a45ff [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6440e49253344afebefac34f6e39e878 (127.13.99.3:40441): Not found: Tablet not found: 9e454f9dfb96467399fa4eb28c8ade3a
I20260812 09:49:08.315768 9456 leader_election.cc:302] T 9e454f9dfb96467399fa4eb28c8ade3a P f9e2dce914b843b2bfc0de5f215a45ff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9e2dce914b843b2bfc0de5f215a45ff; no voters: 6440e49253344afebefac34f6e39e878, 90b746efe2d54317b04ae5164cef1a3b
I20260812 09:49:08.315977 10135 raft_consensus.cc:2726] T 9e454f9dfb96467399fa4eb28c8ade3a P f9e2dce914b843b2bfc0de5f215a45ff [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 09:49:08.480147 10136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:08.479722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:37085 (local address 127.13.99.2:57035)
0812 09:49:08.479985 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:08.480042 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:37085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":79,"threads_started":1}
I20260812 09:49:08.693938 10139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:08.693518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:37085 (local address 127.13.99.3:51265)
0812 09:49:08.693785 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:08.693843 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:37085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":78,"threads_started":1}
I20260812 09:49:09.315090 10141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:09.314663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:37085 (local address 127.13.99.1:44777)
0812 09:49:09.314939 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:09.314992 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:37085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":105,"threads_started":1}
I20260812 09:49:09.481299 10142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:09.480862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:37085 (local address 127.13.99.2:51657)
0812 09:49:09.481153 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:09.481207 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:37085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":90,"threads_started":1}
I20260812 09:49:09.695171 10145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:09.694688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:37085 (local address 127.13.99.3:51749)
0812 09:49:09.695029 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:09.695082 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:37085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":99,"threads_started":1}
I20260812 09:49:10.316414 10147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:10.315867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:37085 (local address 127.13.99.1:60953)
0812 09:49:10.316152 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:10.316218 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:37085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":87,"threads_started":1}
I20260812 09:49:10.482518 10148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:10.482088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:37085 (local address 127.13.99.2:54167)
0812 09:49:10.482346 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:10.482415 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:37085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":78,"threads_started":1}
I20260812 09:49:10.696403 10151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:10.695985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:37085 (local address 127.13.99.3:48725)
0812 09:49:10.696238 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:10.696298 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:37085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":80,"threads_started":1}
I20260812 09:49:11.317682 10153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:11.317241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:37085 (local address 127.13.99.1:57743)
0812 09:49:11.317512 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:11.317579 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:37085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":79,"threads_started":1}
I20260812 09:49:11.337972 10156 raft_consensus.cc:482] T 9e454f9dfb96467399fa4eb28c8ade3a P f9e2dce914b843b2bfc0de5f215a45ff [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 09:49:11.338146 10156 raft_consensus.cc:504] T 9e454f9dfb96467399fa4eb28c8ade3a P f9e2dce914b843b2bfc0de5f215a45ff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b746efe2d54317b04ae5164cef1a3b" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 43737 } } peers { permanent_uuid: "6440e49253344afebefac34f6e39e878" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 40441 } } peers { permanent_uuid: "f9e2dce914b843b2bfc0de5f215a45ff" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 34429 } }
I20260812 09:49:11.338595 10156 leader_election.cc:288] T 9e454f9dfb96467399fa4eb28c8ade3a P f9e2dce914b843b2bfc0de5f215a45ff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 90b746efe2d54317b04ae5164cef1a3b (127.13.99.2:43737), 6440e49253344afebefac34f6e39e878 (127.13.99.3:40441)
W20260812 09:49:11.339483 9452 leader_election.cc:341] T 9e454f9dfb96467399fa4eb28c8ade3a P f9e2dce914b843b2bfc0de5f215a45ff [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 90b746efe2d54317b04ae5164cef1a3b (127.13.99.2:43737): Not found: Tablet not found: 9e454f9dfb96467399fa4eb28c8ade3a
W20260812 09:49:11.339589 9456 leader_election.cc:341] T 9e454f9dfb96467399fa4eb28c8ade3a P f9e2dce914b843b2bfc0de5f215a45ff [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6440e49253344afebefac34f6e39e878 (127.13.99.3:40441): Not found: Tablet not found: 9e454f9dfb96467399fa4eb28c8ade3a
I20260812 09:49:11.339651 9456 leader_election.cc:302] T 9e454f9dfb96467399fa4eb28c8ade3a P f9e2dce914b843b2bfc0de5f215a45ff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9e2dce914b843b2bfc0de5f215a45ff; no voters: 6440e49253344afebefac34f6e39e878, 90b746efe2d54317b04ae5164cef1a3b
I20260812 09:49:11.339787 10156 raft_consensus.cc:2726] T 9e454f9dfb96467399fa4eb28c8ade3a P f9e2dce914b843b2bfc0de5f215a45ff [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 09:49:11.483812 10158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:11.483374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:37085 (local address 127.13.99.2:53483)
0812 09:49:11.483646 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:11.483710 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:37085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":82,"threads_started":1}
I20260812 09:49:11.697592 10159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:11.697166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:37085 (local address 127.13.99.3:50229)
0812 09:49:11.697433 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:11.697498 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:37085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":84,"threads_started":1}
I20260812 09:49:12.319003 10160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:12.318535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:37085 (local address 127.13.99.1:38481)
0812 09:49:12.318835 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:12.318901 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:37085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":86,"threads_started":1}
I20260812 09:49:12.485212 10163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:12.484731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:37085 (local address 127.13.99.2:52591)
0812 09:49:12.485012 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:12.485090 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:37085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":98,"threads_started":1}
I20260812 09:49:12.698948 10164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:12.698488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:37085 (local address 127.13.99.3:53627)
0812 09:49:12.698790 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:12.698851 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:37085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":82,"threads_started":1}
I20260812 09:49:13.320394 10165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:13.319930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:37085 (local address 127.13.99.1:36245)
0812 09:49:13.320229 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:13.320294 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:37085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":83,"threads_started":1}
I20260812 09:49:13.486451 10167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:13.485995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:37085 (local address 127.13.99.2:57685)
0812 09:49:13.486277 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:13.486353 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:37085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":76,"threads_started":1}
I20260812 09:49:13.700241 10172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:13.699763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:37085 (local address 127.13.99.3:40231)
0812 09:49:13.700074 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:13.700143 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:37085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":102,"threads_started":1}
I20260812 09:49:14.321771 10173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:14.321313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:37085 (local address 127.13.99.1:41709)
0812 09:49:14.321613 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:14.321672 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:37085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":124,"threads_started":1}
I20260812 09:49:14.487877 10174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:14.487416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:37085 (local address 127.13.99.2:48973)
0812 09:49:14.487712 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:14.487775 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:37085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":83,"threads_started":1}
I20260812 09:49:14.701570 10177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:14.701092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:37085 (local address 127.13.99.3:48093)
0812 09:49:14.701412 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:14.701471 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:37085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":80,"threads_started":1}
I20260812 09:49:15.322980 10178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:15.322552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:37085 (local address 127.13.99.1:45691)
0812 09:49:15.322823 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:15.322882 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:37085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":76,"threads_started":1}
I20260812 09:49:15.489060 10181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:15.488678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:37085 (local address 127.13.99.2:37131)
0812 09:49:15.488913 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:15.488965 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:37085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":73,"threads_started":1}
I20260812 09:49:15.702819 10182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:15.702390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:37085 (local address 127.13.99.3:38361)
0812 09:49:15.702672 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:15.702728 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:37085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":94,"threads_started":1}
I20260812 09:49:16.324242 10184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:16.323801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:37085 (local address 127.13.99.1:45063)
0812 09:49:16.324072 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:16.324142 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:37085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":79,"threads_started":1}
I20260812 09:49:16.490264 10185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:16.489803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:37085 (local address 127.13.99.2:34863)
0812 09:49:16.490107 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:16.490167 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:37085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":104,"threads_started":1}
I20260812 09:49:16.703990 10188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:16.703561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:37085 (local address 127.13.99.3:49557)
0812 09:49:16.703831 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:16.703888 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:37085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":98,"threads_started":1}
I20260812 09:49:17.325625 10190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:17.325215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:37085 (local address 127.13.99.1:47931)
0812 09:49:17.325468 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:17.325522 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:37085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":73,"threads_started":1}
I20260812 09:49:17.327466 10191 raft_consensus.cc:482] T 9e454f9dfb96467399fa4eb28c8ade3a P f9e2dce914b843b2bfc0de5f215a45ff [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 09:49:17.327579 10191 raft_consensus.cc:504] T 9e454f9dfb96467399fa4eb28c8ade3a P f9e2dce914b843b2bfc0de5f215a45ff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b746efe2d54317b04ae5164cef1a3b" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 43737 } } peers { permanent_uuid: "6440e49253344afebefac34f6e39e878" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 40441 } } peers { permanent_uuid: "f9e2dce914b843b2bfc0de5f215a45ff" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 34429 } }
I20260812 09:49:17.328014 10191 leader_election.cc:288] T 9e454f9dfb96467399fa4eb28c8ade3a P f9e2dce914b843b2bfc0de5f215a45ff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 90b746efe2d54317b04ae5164cef1a3b (127.13.99.2:43737), 6440e49253344afebefac34f6e39e878 (127.13.99.3:40441)
W20260812 09:49:17.328822 9456 leader_election.cc:341] T 9e454f9dfb96467399fa4eb28c8ade3a P f9e2dce914b843b2bfc0de5f215a45ff [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6440e49253344afebefac34f6e39e878 (127.13.99.3:40441): Not found: Tablet not found: 9e454f9dfb96467399fa4eb28c8ade3a
W20260812 09:49:17.328920 9452 leader_election.cc:341] T 9e454f9dfb96467399fa4eb28c8ade3a P f9e2dce914b843b2bfc0de5f215a45ff [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 90b746efe2d54317b04ae5164cef1a3b (127.13.99.2:43737): Not found: Tablet not found: 9e454f9dfb96467399fa4eb28c8ade3a
I20260812 09:49:17.328976 9452 leader_election.cc:302] T 9e454f9dfb96467399fa4eb28c8ade3a P f9e2dce914b843b2bfc0de5f215a45ff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9e2dce914b843b2bfc0de5f215a45ff; no voters: 6440e49253344afebefac34f6e39e878, 90b746efe2d54317b04ae5164cef1a3b
I20260812 09:49:17.329137 10191 raft_consensus.cc:2726] T 9e454f9dfb96467399fa4eb28c8ade3a P f9e2dce914b843b2bfc0de5f215a45ff [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 09:49:17.491554 10193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:17.491127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:37085 (local address 127.13.99.2:49475)
0812 09:49:17.491395 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:17.491457 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:37085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":73,"threads_started":1}
I20260812 09:49:17.691609 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 9357
I20260812 09:49:17.705559 10194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:17.705109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:37085 (local address 127.13.99.3:44955)
0812 09:49:17.705399 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:17.705458 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.62:37085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":104,"threads_started":1}
I20260812 09:49:17.707022 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 9664
I20260812 09:49:17.719197 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 9881
2026-08-12T09:49:17Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59687 ms)
[----------] 30 tests from SecurityITest (251796 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 09:49:17 dist-test-slave-jq4t krb5kdc[10197](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 09:49:17 dist-test-slave-jq4t krb5kdc[10197](info): set up 2 sockets
Aug 12 09:49:17 dist-test-slave-jq4t krb5kdc[10197](info): commencing operation
krb5kdc: starting...
W20260812 09:49:19.023778 13708 mini_kdc.cc:121] Time spent starting KDC: real 1.259s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/krb5kdc/test-user.keytab.
Aug 12 09:49:19 dist-test-slave-jq4t krb5kdc[10197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T09:49:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T09:49:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.99.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.99.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.99.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab.
Entry for principal kudu/127.13.99.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab.
I20260812 09:49:19.153262 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.62:39585
--webserver_interface=127.13.99.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab
--principal=kudu/127.13.99.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:39069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.99.62:39585
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:49:19.266611 10219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:49:19.266954 10219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:49:19.267042 10219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:49:19.271028 10219 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 09:49:19.271116 10219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:49:19.271155 10219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:49:19.271214 10219 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 09:49:19.271255 10219 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 09:49:19.276407 10219 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:39069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.99.62:39585
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab
--principal=kudu/127.13.99.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.62:39585
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.99.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10219
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:49:19.277438 10219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:49:19.278432 10219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:49:19.287333 10225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:49:19.287348 10229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:49:19.287425 10226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:49:19.287626 10219 server_base.cc:1034] running on GCE node
I20260812 09:49:19.288199 10219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:49:19.289328 10219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:49:19.290503 10219 hybrid_clock.cc:648] HybridClock initialized: now 1786528159290484 us; error 30 us; skew 500 ppm
Aug 12 09:49:19 dist-test-slave-jq4t krb5kdc[10197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528159, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:49:19.293354 10219 init.cc:377] Logged in from keytab as kudu/127.13.99.62@KRBTEST.COM (short username kudu)
I20260812 09:49:19.294281 10219 webserver.cc:460] Webserver started at http://127.13.99.62:33755/ using document root <none> and password file <none>
I20260812 09:49:19.294838 10219 fs_manager.cc:359] Metadata directory not provided
I20260812 09:49:19.294935 10219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:49:19.295174 10219 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:49:19.297219 10219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/minicluster-data/master-0/data/instance:
uuid: "6be03e025f384bafbdaddc22a849f405"
format_stamp: "Formatted at 2026-08-12 09:49:19 on dist-test-slave-jq4t"
server_key: "87b12fc7579d72d06d53925a6ca83c5d"
server_key_iv: "7a6337645e582e9f880a4720f815a0e4"
server_key_version: "encryptionkey@0"
I20260812 09:49:19.297788 10219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/minicluster-data/master-0/wal/instance:
uuid: "6be03e025f384bafbdaddc22a849f405"
format_stamp: "Formatted at 2026-08-12 09:49:19 on dist-test-slave-jq4t"
server_key: "87b12fc7579d72d06d53925a6ca83c5d"
server_key_iv: "7a6337645e582e9f880a4720f815a0e4"
server_key_version: "encryptionkey@0"
I20260812 09:49:19.301510 10219 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260812 09:49:19.303841 10242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:49:19.305096 10219 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260812 09:49:19.305250 10219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/minicluster-data/master-0/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/minicluster-data/master-0/wal
uuid: "6be03e025f384bafbdaddc22a849f405"
format_stamp: "Formatted at 2026-08-12 09:49:19 on dist-test-slave-jq4t"
server_key: "87b12fc7579d72d06d53925a6ca83c5d"
server_key_iv: "7a6337645e582e9f880a4720f815a0e4"
server_key_version: "encryptionkey@0"
I20260812 09:49:19.305361 10219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:49:19.336750 10219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:49:19.339787 10219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:49:19.340004 10219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:49:19.347309 10219 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.62:39585
I20260812 09:49:19.347337 10319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.62:39585 every 8 connection(s)
I20260812 09:49:19.348246 10219 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
I20260812 09:49:19.349640 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 10219
I20260812 09:49:19.349777 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/minicluster-data/master-0/wal/instance
I20260812 09:49:19.350082 13708 external_mini_cluster.cc:1428] Setting key ad9b05ed7db758fa4779b87046821677
I20260812 09:49:19.351305 10320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:49:19.357821 10320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6be03e025f384bafbdaddc22a849f405: Bootstrap starting.
Aug 12 09:49:19 dist-test-slave-jq4t krb5kdc[10197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
I20260812 09:49:19.360263 10320 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6be03e025f384bafbdaddc22a849f405: Neither blocks nor log segments found. Creating new log.
I20260812 09:49:19.361042 10320 log.cc:824] T 00000000000000000000000000000000 P 6be03e025f384bafbdaddc22a849f405: Log is configured to *not* fsync() on all Append() calls
I20260812 09:49:19.362826 10320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6be03e025f384bafbdaddc22a849f405: No bootstrap required, opened a new log
I20260812 09:49:19.364457 10326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:19.351538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38872 (local address 127.13.99.62:39585)
0812 09:49:19.351953 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:19.351972 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:19.352015 (+ 43us) server_negotiation.cc:407] Connection header received
0812 09:49:19.352734 (+ 719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:19.352752 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:19.353057 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:19.353316 (+ 259us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:49:19.354246 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:19.355258 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:19.356436 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:19.356949 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:19.357072 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:19.359766 (+ 2694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:49:19.359771 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:49:19.359784 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:49:19.359811 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:49:19.361698 (+ 1887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:49:19.362138 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:49:19.362146 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:49:19.362156 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:49:19.362263 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:49:19.362578 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:49:19.362581 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:49:19.362583 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:49:19.363007 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:49:19.363201 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:19.363506 (+ 305us) server_negotiation.cc:299] Negotiation successful
0812 09:49:19.363794 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":142,"threads_started":1}
I20260812 09:49:19.367238 10320 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6be03e025f384bafbdaddc22a849f405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6be03e025f384bafbdaddc22a849f405" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 39585 } }
I20260812 09:49:19.367453 10320 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6be03e025f384bafbdaddc22a849f405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:49:19.367497 10320 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6be03e025f384bafbdaddc22a849f405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6be03e025f384bafbdaddc22a849f405, State: Initialized, Role: FOLLOWER
I20260812 09:49:19.367841 10320 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6be03e025f384bafbdaddc22a849f405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6be03e025f384bafbdaddc22a849f405" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 39585 } }
I20260812 09:49:19.367952 10320 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6be03e025f384bafbdaddc22a849f405 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 09:49:19.367995 10320 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6be03e025f384bafbdaddc22a849f405 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 09:49:19.368063 10320 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6be03e025f384bafbdaddc22a849f405 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:49:19.368999 10320 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6be03e025f384bafbdaddc22a849f405 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6be03e025f384bafbdaddc22a849f405" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 39585 } }
I20260812 09:49:19.369289 10320 leader_election.cc:302] T 00000000000000000000000000000000 P 6be03e025f384bafbdaddc22a849f405 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6be03e025f384bafbdaddc22a849f405; no voters:
I20260812 09:49:19.369572 10320 leader_election.cc:288] T 00000000000000000000000000000000 P 6be03e025f384bafbdaddc22a849f405 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 09:49:19.369728 10331 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6be03e025f384bafbdaddc22a849f405 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 09:49:19.370002 10331 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6be03e025f384bafbdaddc22a849f405 [term 1 LEADER]: Becoming Leader. State: Replica: 6be03e025f384bafbdaddc22a849f405, State: Running, Role: LEADER
I20260812 09:49:19.370370 10331 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6be03e025f384bafbdaddc22a849f405 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6be03e025f384bafbdaddc22a849f405" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 39585 } }
I20260812 09:49:19.370916 10320 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6be03e025f384bafbdaddc22a849f405 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 09:49:19.372156 10336 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6be03e025f384bafbdaddc22a849f405 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6be03e025f384bafbdaddc22a849f405. Latest consensus state: current_term: 1 leader_uuid: "6be03e025f384bafbdaddc22a849f405" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6be03e025f384bafbdaddc22a849f405" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 39585 } } }
I20260812 09:49:19.372288 10336 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6be03e025f384bafbdaddc22a849f405 [sys.catalog]: This master's current role is: LEADER
I20260812 09:49:19.372697 10344 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 09:49:19.372681 10335 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6be03e025f384bafbdaddc22a849f405 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6be03e025f384bafbdaddc22a849f405" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6be03e025f384bafbdaddc22a849f405" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 39585 } } }
I20260812 09:49:19.373435 10335 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6be03e025f384bafbdaddc22a849f405 [sys.catalog]: This master's current role is: LEADER
I20260812 09:49:19.375937 10344 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 09:49:19.381469 10344 catalog_manager.cc:1347] Generated new cluster ID: 7bca284d227e4b4c86157ab04032dd60
I20260812 09:49:19.381534 10344 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 09:49:19.400385 10344 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 09:49:19.401333 10344 catalog_manager.cc:1504] Loading token signing keys...
I20260812 09:49:19.406543 10344 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6be03e025f384bafbdaddc22a849f405: Generated new TSK 0
I20260812 09:49:19.407184 10344 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.99.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.99.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.99.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.1.keytab.
Entry for principal kudu/127.13.99.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.1.keytab.
I20260812 09:49:19.444012 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.99.1:0
--local_ip_for_outbound_sockets=127.13.99.1
--webserver_interface=127.13.99.1
--webserver_port=0
--tserver_master_addrs=127.13.99.62:39585
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.1.keytab
--principal=kudu/127.13.99.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:39069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:49:19.562071 10363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:49:19.562317 10363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:49:19.562417 10363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:49:19.566545 10363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:49:19.566733 10363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:49:19.566901 10363 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.99.1
I20260812 09:49:19.572561 10363 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:39069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.1.keytab
--principal=kudu/127.13.99.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.99.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.99.1
--webserver_port=0
--tserver_master_addrs=127.13.99.62:39585
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10363
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.99.1
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:49:19.573693 10363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:49:19.574728 10363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:49:19.584757 10371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:49:19.584748 10374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:49:19.584743 10370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:49:19.585067 10363 server_base.cc:1034] running on GCE node
I20260812 09:49:19.585602 10363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:49:19.586187 10363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:49:19.587355 10363 hybrid_clock.cc:648] HybridClock initialized: now 1786528159587337 us; error 27 us; skew 500 ppm
Aug 12 09:49:19 dist-test-slave-jq4t krb5kdc[10197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528159, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:49:19.590272 10363 init.cc:377] Logged in from keytab as kudu/127.13.99.1@KRBTEST.COM (short username kudu)
I20260812 09:49:19.591238 10363 webserver.cc:460] Webserver started at http://127.13.99.1:45189/ using document root <none> and password file <none>
I20260812 09:49:19.591889 10363 fs_manager.cc:359] Metadata directory not provided
I20260812 09:49:19.591972 10363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:49:19.592195 10363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:49:19.594306 10363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/minicluster-data/ts-0/data/instance:
uuid: "cb0c589ba2fd456b8ac0377d4d81800b"
format_stamp: "Formatted at 2026-08-12 09:49:19 on dist-test-slave-jq4t"
server_key: "df0ec23f78c7165c1ffc91083ebd29f0"
server_key_iv: "d8a102b0373aea22c7129fa594f30b0f"
server_key_version: "encryptionkey@0"
I20260812 09:49:19.594894 10363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/minicluster-data/ts-0/wal/instance:
uuid: "cb0c589ba2fd456b8ac0377d4d81800b"
format_stamp: "Formatted at 2026-08-12 09:49:19 on dist-test-slave-jq4t"
server_key: "df0ec23f78c7165c1ffc91083ebd29f0"
server_key_iv: "d8a102b0373aea22c7129fa594f30b0f"
server_key_version: "encryptionkey@0"
I20260812 09:49:19.598687 10363 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 09:49:19.601272 10387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:49:19.602620 10363 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 09:49:19.602782 10363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/minicluster-data/ts-0/wal
uuid: "cb0c589ba2fd456b8ac0377d4d81800b"
format_stamp: "Formatted at 2026-08-12 09:49:19 on dist-test-slave-jq4t"
server_key: "df0ec23f78c7165c1ffc91083ebd29f0"
server_key_iv: "d8a102b0373aea22c7129fa594f30b0f"
server_key_version: "encryptionkey@0"
I20260812 09:49:19.602905 10363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:49:19.625010 10363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:49:19.628408 10363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:49:19.628657 10363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:49:19.629302 10363 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 09:49:19.630196 10363 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:49:19.630270 10363 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 09:49:19.630348 10363 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:49:19.630404 10363 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 09:49:19.640470 10363 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.1:43379
I20260812 09:49:19.640641 10576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.1:43379 every 8 connection(s)
I20260812 09:49:19.641505 10363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/minicluster-data/ts-0/data/info.pb
Aug 12 09:49:19 dist-test-slave-jq4t krb5kdc[10197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528159, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.1@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
I20260812 09:49:19.651306 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 10363
I20260812 09:49:19.651446 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/minicluster-data/ts-0/wal/instance
I20260812 09:49:19.651739 13708 external_mini_cluster.cc:1428] Setting key f524e81552ed3c7635d6bb22149703da
I20260812 09:49:19.655352 10326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:19.643473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.1:33199 (local address 127.13.99.62:39585)
0812 09:49:19.643655 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:19.643661 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:19.644734 (+ 1073us) server_negotiation.cc:407] Connection header received
0812 09:49:19.645599 (+ 865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:19.645602 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:19.645651 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:19.645728 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:49:19.646916 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:19.647473 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:19.648717 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:19.648967 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:19.649060 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:19.651613 (+ 2553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:49:19.651617 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:49:19.651619 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:49:19.651643 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:49:19.653316 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:49:19.653834 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:49:19.653838 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:49:19.653839 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:49:19.653923 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:49:19.654274 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:49:19.654278 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:49:19.654279 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:49:19.654462 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:49:19.654568 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:19.655056 (+ 488us) server_negotiation.cc:299] Negotiation successful
0812 09:49:19.655184 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260812 09:49:19.656138 10585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:19.643868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:39585 (local address 127.13.99.1:33199)
0812 09:49:19.644590 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:19.644635 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 09:49:19.645396 (+ 761us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:49:19.645898 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:49:19.645904 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:49:19.646254 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:49:19.646753 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:49:19.646766 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:49:19.647593 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:49:19.647596 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:49:19.648598 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:49:19.648609 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:49:19.649116 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:49:19.649119 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:49:19.649271 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:19.649879 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:49:19.649895 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:49:19.651412 (+ 1517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:49:19.653429 (+ 2017us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:49:19.653435 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:49:19.653450 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:49:19.653717 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:49:19.654017 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:49:19.654021 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:49:19.654023 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:49:19.654166 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:49:19.654567 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:49:19.654575 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:49:19.654823 (+ 248us) client_negotiation.cc:769] Sending connection context
0812 09:49:19.655012 (+ 189us) client_negotiation.cc:240] Negotiation successful
0812 09:49:19.655333 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":103,"threads_started":1}
I20260812 09:49:19.657503 10577 heartbeater.cc:347] Connected to a master server at 127.13.99.62:39585
I20260812 09:49:19.657816 10577 heartbeater.cc:464] Registering TS with master...
I20260812 09:49:19.658345 10577 heartbeater.cc:511] Master 127.13.99.62:39585 requested a full tablet report, sending...
I20260812 09:49:19.660118 10264 ts_manager.cc:194] Registered new tserver with Master: cb0c589ba2fd456b8ac0377d4d81800b (127.13.99.1:43379)
I20260812 09:49:19.661875 10264 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.99.1@KRBTEST.COM'} at 127.13.99.1:33199
WARNING: no policy specified for kudu/127.13.99.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.99.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.99.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.2.keytab.
Entry for principal kudu/127.13.99.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.2.keytab.
I20260812 09:49:19.683691 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.99.2:0
--local_ip_for_outbound_sockets=127.13.99.2
--webserver_interface=127.13.99.2
--webserver_port=0
--tserver_master_addrs=127.13.99.62:39585
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.2.keytab
--principal=kudu/127.13.99.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:39069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:49:19.799238 10590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:49:19.799508 10590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:49:19.799553 10590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:49:19.803691 10590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:49:19.803747 10590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:49:19.803805 10590 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.99.2
I20260812 09:49:19.809103 10590 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:39069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.2.keytab
--principal=kudu/127.13.99.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.99.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.99.2
--webserver_port=0
--tserver_master_addrs=127.13.99.62:39585
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10590
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.99.2
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:49:19.810165 10590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:49:19.811167 10590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:49:19.820860 10600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:49:19.820892 10603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:49:19.820997 10590 server_base.cc:1034] running on GCE node
W20260812 09:49:19.821084 10599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:49:19.821563 10590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:49:19.822135 10590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:49:19.823308 10590 hybrid_clock.cc:648] HybridClock initialized: now 1786528159823295 us; error 22 us; skew 500 ppm
Aug 12 09:49:19 dist-test-slave-jq4t krb5kdc[10197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528159, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:49:19.826171 10590 init.cc:377] Logged in from keytab as kudu/127.13.99.2@KRBTEST.COM (short username kudu)
I20260812 09:49:19.827091 10590 webserver.cc:460] Webserver started at http://127.13.99.2:45201/ using document root <none> and password file <none>
I20260812 09:49:19.827662 10590 fs_manager.cc:359] Metadata directory not provided
I20260812 09:49:19.827723 10590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:49:19.827952 10590 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:49:19.829993 10590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/minicluster-data/ts-1/data/instance:
uuid: "28952c7c86c640a3877ad743d4d8ac7f"
format_stamp: "Formatted at 2026-08-12 09:49:19 on dist-test-slave-jq4t"
server_key: "1650506ebc05d0d6936ba42369de5b4f"
server_key_iv: "e0ccf13f44bc783738bff12e55570708"
server_key_version: "encryptionkey@0"
I20260812 09:49:19.830583 10590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/minicluster-data/ts-1/wal/instance:
uuid: "28952c7c86c640a3877ad743d4d8ac7f"
format_stamp: "Formatted at 2026-08-12 09:49:19 on dist-test-slave-jq4t"
server_key: "1650506ebc05d0d6936ba42369de5b4f"
server_key_iv: "e0ccf13f44bc783738bff12e55570708"
server_key_version: "encryptionkey@0"
I20260812 09:49:19.834156 10590 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260812 09:49:19.836447 10615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:49:19.837754 10590 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 09:49:19.837916 10590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/minicluster-data/ts-1/wal
uuid: "28952c7c86c640a3877ad743d4d8ac7f"
format_stamp: "Formatted at 2026-08-12 09:49:19 on dist-test-slave-jq4t"
server_key: "1650506ebc05d0d6936ba42369de5b4f"
server_key_iv: "e0ccf13f44bc783738bff12e55570708"
server_key_version: "encryptionkey@0"
I20260812 09:49:19.838032 10590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:49:19.850924 10590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:49:19.853662 10590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:49:19.853857 10590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:49:19.854413 10590 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 09:49:19.855191 10590 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:49:19.855261 10590 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 09:49:19.855329 10590 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:49:19.855391 10590 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 09:49:19.864562 10590 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.2:35377
I20260812 09:49:19.864580 10804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.2:35377 every 8 connection(s)
I20260812 09:49:19.865442 10590 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/minicluster-data/ts-1/data/info.pb
I20260812 09:49:19.871133 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 10590
I20260812 09:49:19.871249 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/minicluster-data/ts-1/wal/instance
I20260812 09:49:19.871563 13708 external_mini_cluster.cc:1428] Setting key 3c7a7a44962ffafcb9418e0943f47165
Aug 12 09:49:19 dist-test-slave-jq4t krb5kdc[10197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528159, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.2@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
I20260812 09:49:19.879739 10326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:19.867183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.2:55715 (local address 127.13.99.62:39585)
0812 09:49:19.867309 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:19.867320 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:19.868282 (+ 962us) server_negotiation.cc:407] Connection header received
0812 09:49:19.869222 (+ 940us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:19.869225 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:19.869287 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:19.869370 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:49:19.870557 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:19.871339 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:19.872919 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:19.873295 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:19.873408 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:19.876171 (+ 2763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:49:19.876175 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:49:19.876177 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:49:19.876210 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:49:19.877709 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:49:19.878238 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:49:19.878246 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:49:19.878247 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:49:19.878321 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:49:19.878674 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:49:19.878679 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:49:19.878681 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:49:19.878864 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:49:19.878982 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:19.879462 (+ 480us) server_negotiation.cc:299] Negotiation successful
0812 09:49:19.879582 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260812 09:49:19.880669 10808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:19.867491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:39585 (local address 127.13.99.2:55715)
0812 09:49:19.868139 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:19.868178 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0812 09:49:19.869017 (+ 839us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:49:19.869564 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:49:19.869571 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:49:19.869903 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:49:19.870406 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:49:19.870419 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:49:19.871471 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:49:19.871474 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:49:19.872784 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:49:19.872796 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:49:19.873445 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:49:19.873448 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:49:19.873664 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:19.874284 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:49:19.874301 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:49:19.875967 (+ 1666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:49:19.877833 (+ 1866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:49:19.877840 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:49:19.877855 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:49:19.878120 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:49:19.878417 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:49:19.878420 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:49:19.878422 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:49:19.878574 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:49:19.878974 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:49:19.878981 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:49:19.879227 (+ 246us) client_negotiation.cc:769] Sending connection context
0812 09:49:19.879418 (+ 191us) client_negotiation.cc:240] Negotiation successful
0812 09:49:19.879742 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":120,"threads_started":1}
I20260812 09:49:19.881997 10805 heartbeater.cc:347] Connected to a master server at 127.13.99.62:39585
I20260812 09:49:19.882237 10805 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.13.99.3@KRBTEST.COM; defaulting to no policy
I20260812 09:49:19.882761 10805 heartbeater.cc:511] Master 127.13.99.62:39585 requested a full tablet report, sending...
I20260812 09:49:19.883898 10264 ts_manager.cc:194] Registered new tserver with Master: 28952c7c86c640a3877ad743d4d8ac7f (127.13.99.2:35377)
I20260812 09:49:19.884610 10264 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.99.2@KRBTEST.COM'} at 127.13.99.2:55715
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.99.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.99.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.3.keytab.
Entry for principal kudu/127.13.99.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.3.keytab.
I20260812 09:49:19.901895 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.99.3:0
--local_ip_for_outbound_sockets=127.13.99.3
--webserver_interface=127.13.99.3
--webserver_port=0
--tserver_master_addrs=127.13.99.62:39585
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.3.keytab
--principal=kudu/127.13.99.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:39069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:49:20.015882 10811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:49:20.016136 10811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:49:20.016178 10811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:49:20.020278 10811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:49:20.020341 10811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:49:20.020399 10811 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.99.3
I20260812 09:49:20.025619 10811 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:39069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.3.keytab
--principal=kudu/127.13.99.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.99.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.99.3
--webserver_port=0
--tserver_master_addrs=127.13.99.62:39585
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10811
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.99.3
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:49:20.026713 10811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:49:20.027791 10811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:49:20.037638 10819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:49:20.037626 10818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:49:20.037678 10824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:49:20.037949 10811 server_base.cc:1034] running on GCE node
I20260812 09:49:20.038470 10811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:49:20.039100 10811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:49:20.040293 10811 hybrid_clock.cc:648] HybridClock initialized: now 1786528160040270 us; error 38 us; skew 500 ppm
Aug 12 09:49:20 dist-test-slave-jq4t krb5kdc[10197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528160, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:49:20.043179 10811 init.cc:377] Logged in from keytab as kudu/127.13.99.3@KRBTEST.COM (short username kudu)
I20260812 09:49:20.044030 10811 webserver.cc:460] Webserver started at http://127.13.99.3:37175/ using document root <none> and password file <none>
I20260812 09:49:20.044680 10811 fs_manager.cc:359] Metadata directory not provided
I20260812 09:49:20.044775 10811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:49:20.045011 10811 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:49:20.047040 10811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/minicluster-data/ts-2/data/instance:
uuid: "bfa6698e37504e508c1ea2f09791f1c8"
format_stamp: "Formatted at 2026-08-12 09:49:20 on dist-test-slave-jq4t"
server_key: "f75edcdae8e2a8ee5173c83204aa31ba"
server_key_iv: "69cb1b2cae9d6b1524261422bdcf3ea4"
server_key_version: "encryptionkey@0"
I20260812 09:49:20.047636 10811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/minicluster-data/ts-2/wal/instance:
uuid: "bfa6698e37504e508c1ea2f09791f1c8"
format_stamp: "Formatted at 2026-08-12 09:49:20 on dist-test-slave-jq4t"
server_key: "f75edcdae8e2a8ee5173c83204aa31ba"
server_key_iv: "69cb1b2cae9d6b1524261422bdcf3ea4"
server_key_version: "encryptionkey@0"
I20260812 09:49:20.051407 10811 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260812 09:49:20.053879 10834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:49:20.055101 10811 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260812 09:49:20.055238 10811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/minicluster-data/ts-2/wal
uuid: "bfa6698e37504e508c1ea2f09791f1c8"
format_stamp: "Formatted at 2026-08-12 09:49:20 on dist-test-slave-jq4t"
server_key: "f75edcdae8e2a8ee5173c83204aa31ba"
server_key_iv: "69cb1b2cae9d6b1524261422bdcf3ea4"
server_key_version: "encryptionkey@0"
I20260812 09:49:20.055349 10811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:49:20.075443 10811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:49:20.078402 10811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:49:20.078616 10811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:49:20.079227 10811 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 09:49:20.080078 10811 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:49:20.080152 10811 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 09:49:20.080217 10811 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:49:20.080260 10811 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 09:49:20.089648 10811 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.3:36967
I20260812 09:49:20.089675 11019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.3:36967 every 8 connection(s)
I20260812 09:49:20.090560 10811 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/minicluster-data/ts-2/data/info.pb
I20260812 09:49:20.099509 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 10811
I20260812 09:49:20.099669 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/minicluster-data/ts-2/wal/instance
I20260812 09:49:20.099946 13708 external_mini_cluster.cc:1428] Setting key dd74f6f0c2c882c47b59e2182e801b90
Aug 12 09:49:20 dist-test-slave-jq4t krb5kdc[10197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528160, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.3@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
I20260812 09:49:20.104305 10326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:20.092401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.3:57569 (local address 127.13.99.62:39585)
0812 09:49:20.092581 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:20.092585 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:20.093679 (+ 1094us) server_negotiation.cc:407] Connection header received
0812 09:49:20.094619 (+ 940us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:20.094622 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:20.094684 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:20.094786 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:49:20.095970 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:20.096592 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:20.097730 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:20.097966 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:20.098077 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:20.100883 (+ 2806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:49:20.100887 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:49:20.100890 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:49:20.100925 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:49:20.102537 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:49:20.103001 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:49:20.103004 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:49:20.103005 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:49:20.103092 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:49:20.103388 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:49:20.103390 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:49:20.103391 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:49:20.103565 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:49:20.103639 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:20.104100 (+ 461us) server_negotiation.cc:299] Negotiation successful
0812 09:49:20.104193 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260812 09:49:20.105096 11023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:20.092805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:39585 (local address 127.13.99.3:57569)
0812 09:49:20.093516 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:20.093560 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 09:49:20.094383 (+ 823us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:49:20.094962 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:49:20.094969 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:49:20.095335 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:49:20.095815 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:49:20.095829 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:49:20.096717 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:49:20.096727 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:49:20.097595 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:49:20.097604 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:49:20.098118 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:49:20.098122 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:49:20.098291 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:20.098853 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:49:20.098869 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:49:20.100687 (+ 1818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:49:20.102643 (+ 1956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:49:20.102647 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:49:20.102658 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:49:20.102913 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:49:20.103175 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:49:20.103177 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:49:20.103178 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:49:20.103308 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:49:20.103654 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:49:20.103658 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:49:20.103874 (+ 216us) client_negotiation.cc:769] Sending connection context
0812 09:49:20.104035 (+ 161us) client_negotiation.cc:240] Negotiation successful
0812 09:49:20.104269 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":131,"threads_started":1}
I20260812 09:49:20.106228 11021 heartbeater.cc:347] Connected to a master server at 127.13.99.62:39585
I20260812 09:49:20.106468 11021 heartbeater.cc:464] Registering TS with master...
I20260812 09:49:20.106990 11021 heartbeater.cc:511] Master 127.13.99.62:39585 requested a full tablet report, sending...
I20260812 09:49:20.108057 10264 ts_manager.cc:194] Registered new tserver with Master: bfa6698e37504e508c1ea2f09791f1c8 (127.13.99.3:36967)
I20260812 09:49:20.108726 10264 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.99.3@KRBTEST.COM'} at 127.13.99.3:57569
I20260812 09:49:20.113816 13708 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 09:49:20.123095 10326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:20.115739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38878 (local address 127.13.99.62:39585)
0812 09:49:20.115863 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:20.115867 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:20.116107 (+ 240us) server_negotiation.cc:407] Connection header received
0812 09:49:20.116255 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:20.116259 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:20.116301 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:20.116390 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:49:20.117101 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:20.117596 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:20.118442 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:20.118669 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:20.118775 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:20.120141 (+ 1366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:49:20.120144 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:49:20.120147 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:49:20.120172 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:49:20.121392 (+ 1220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:49:20.121847 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:49:20.121856 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:49:20.121858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:49:20.121931 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:49:20.122217 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:49:20.122220 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:49:20.122222 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:49:20.122431 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:49:20.122575 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:20.122810 (+ 235us) server_negotiation.cc:299] Negotiation successful
0812 09:49:20.122919 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260812 09:49:20.125806 10264 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38878:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 09:49:20.128895 10264 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 09:49:20.143302 11044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:20.137252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.1:43379 (local address 127.0.0.1:55980)
0812 09:49:20.138041 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:20.138119 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0812 09:49:20.138248 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:49:20.139073 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:49:20.139076 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:49:20.139117 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:49:20.139255 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:49:20.139261 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:49:20.140381 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:49:20.140386 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:49:20.141688 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:49:20.141700 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:49:20.142749 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:49:20.142753 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:49:20.142871 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:20.142918 (+ 47us) client_negotiation.cc:769] Sending connection context
0812 09:49:20.143006 (+ 88us) client_negotiation.cc:240] Negotiation successful
0812 09:49:20.143160 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":727,"mutex_wait_us":2,"spinlock_wait_cycles":2432,"thread_start_us":106,"threads_started":1}
I20260812 09:49:20.143586 11037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:20.137532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.3:36967 (local address 127.0.0.1:34356)
0812 09:49:20.138463 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:20.138476 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 09:49:20.138617 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:49:20.139239 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:49:20.139242 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:49:20.139256 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:49:20.139356 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:49:20.139363 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:49:20.140839 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:49:20.140843 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:49:20.142131 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:49:20.142144 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:49:20.143167 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:49:20.143171 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:49:20.143256 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:20.143292 (+ 36us) client_negotiation.cc:769] Sending connection context
0812 09:49:20.143366 (+ 74us) client_negotiation.cc:240] Negotiation successful
0812 09:49:20.143437 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":868,"mutex_wait_us":31,"thread_start_us":98,"threads_started":1}
I20260812 09:49:20.143638 11046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:20.137942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34356 (local address 127.13.99.3:36967)
0812 09:49:20.138370 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:20.138384 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:20.138552 (+ 168us) server_negotiation.cc:407] Connection header received
0812 09:49:20.138756 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:20.138761 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:20.138907 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:20.139073 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:49:20.139480 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:20.140677 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:20.142257 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:20.143072 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:20.143168 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:20.143300 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:20.143398 (+ 98us) server_negotiation.cc:299] Negotiation successful
0812 09:49:20.143500 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":142,"threads_started":1}
I20260812 09:49:20.143786 11045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:20.137936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55980 (local address 127.13.99.1:43379)
0812 09:49:20.138573 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:20.138590 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:20.138609 (+ 19us) server_negotiation.cc:407] Connection header received
0812 09:49:20.138697 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:20.138704 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:20.138903 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:20.139410 (+ 507us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:49:20.139566 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:20.140253 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:20.141801 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:20.142606 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:20.143325 (+ 719us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:20.143475 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:20.143525 (+ 50us) server_negotiation.cc:299] Negotiation successful
0812 09:49:20.143613 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":97,"threads_started":1}
I20260812 09:49:20.144421 11040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:20.137143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.2:35377 (local address 127.0.0.1:33260)
0812 09:49:20.138446 (+ 1303us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:20.138463 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 09:49:20.138567 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:49:20.139069 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:49:20.139076 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:49:20.139098 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:49:20.139247 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:49:20.139254 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:49:20.140839 (+ 1585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:49:20.140843 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:49:20.142182 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:49:20.142193 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:49:20.143066 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:49:20.143070 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:49:20.144137 (+ 1067us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:20.144161 (+ 24us) client_negotiation.cc:769] Sending connection context
0812 09:49:20.144237 (+ 76us) client_negotiation.cc:240] Negotiation successful
0812 09:49:20.144302 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"mutex_wait_us":992,"thread_start_us":71,"threads_started":1}
I20260812 09:49:20.144985 11042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:20.137277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33260 (local address 127.13.99.2:35377)
0812 09:49:20.138418 (+ 1141us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:20.138430 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:20.138664 (+ 234us) server_negotiation.cc:407] Connection header received
0812 09:49:20.138757 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:20.138760 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:20.138903 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:20.139067 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:49:20.139372 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:20.140676 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:20.142299 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:20.142934 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:20.143070 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:20.143224 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:20.144756 (+ 1532us) server_negotiation.cc:299] Negotiation successful
0812 09:49:20.144855 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1030,"thread_start_us":137,"threads_started":1}
I20260812 09:49:20.146952 10475 tablet_service.cc:1467] Processing CreateTablet for tablet c107931f64ca44248f95222c4b8c039b (DEFAULT_TABLE table=test-table [id=5340ffbff22545608ee50b947d6d3385]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 09:49:20.147187 10692 tablet_service.cc:1467] Processing CreateTablet for tablet c107931f64ca44248f95222c4b8c039b (DEFAULT_TABLE table=test-table [id=5340ffbff22545608ee50b947d6d3385]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 09:49:20.147269 10919 tablet_service.cc:1467] Processing CreateTablet for tablet c107931f64ca44248f95222c4b8c039b (DEFAULT_TABLE table=test-table [id=5340ffbff22545608ee50b947d6d3385]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 09:49:20.148144 10919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c107931f64ca44248f95222c4b8c039b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:49:20.148144 10475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c107931f64ca44248f95222c4b8c039b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:49:20.148144 10692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c107931f64ca44248f95222c4b8c039b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:49:20.154232 11048 tablet_bootstrap.cc:492] T c107931f64ca44248f95222c4b8c039b P 28952c7c86c640a3877ad743d4d8ac7f: Bootstrap starting.
I20260812 09:49:20.155666 11050 tablet_bootstrap.cc:492] T c107931f64ca44248f95222c4b8c039b P cb0c589ba2fd456b8ac0377d4d81800b: Bootstrap starting.
I20260812 09:49:20.156749 11048 tablet_bootstrap.cc:654] T c107931f64ca44248f95222c4b8c039b P 28952c7c86c640a3877ad743d4d8ac7f: Neither blocks nor log segments found. Creating new log.
I20260812 09:49:20.156841 11051 tablet_bootstrap.cc:492] T c107931f64ca44248f95222c4b8c039b P bfa6698e37504e508c1ea2f09791f1c8: Bootstrap starting.
I20260812 09:49:20.157836 11050 tablet_bootstrap.cc:654] T c107931f64ca44248f95222c4b8c039b P cb0c589ba2fd456b8ac0377d4d81800b: Neither blocks nor log segments found. Creating new log.
I20260812 09:49:20.158052 11048 log.cc:824] T c107931f64ca44248f95222c4b8c039b P 28952c7c86c640a3877ad743d4d8ac7f: Log is configured to *not* fsync() on all Append() calls
I20260812 09:49:20.158646 11050 log.cc:824] T c107931f64ca44248f95222c4b8c039b P cb0c589ba2fd456b8ac0377d4d81800b: Log is configured to *not* fsync() on all Append() calls
I20260812 09:49:20.158987 11051 tablet_bootstrap.cc:654] T c107931f64ca44248f95222c4b8c039b P bfa6698e37504e508c1ea2f09791f1c8: Neither blocks nor log segments found. Creating new log.
I20260812 09:49:20.160252 11051 log.cc:824] T c107931f64ca44248f95222c4b8c039b P bfa6698e37504e508c1ea2f09791f1c8: Log is configured to *not* fsync() on all Append() calls
I20260812 09:49:20.160827 11048 tablet_bootstrap.cc:492] T c107931f64ca44248f95222c4b8c039b P 28952c7c86c640a3877ad743d4d8ac7f: No bootstrap required, opened a new log
I20260812 09:49:20.161074 11048 ts_tablet_manager.cc:1397] T c107931f64ca44248f95222c4b8c039b P 28952c7c86c640a3877ad743d4d8ac7f: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.001s
I20260812 09:49:20.161696 11050 tablet_bootstrap.cc:492] T c107931f64ca44248f95222c4b8c039b P cb0c589ba2fd456b8ac0377d4d81800b: No bootstrap required, opened a new log
I20260812 09:49:20.161908 11050 ts_tablet_manager.cc:1397] T c107931f64ca44248f95222c4b8c039b P cb0c589ba2fd456b8ac0377d4d81800b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260812 09:49:20.162357 11051 tablet_bootstrap.cc:492] T c107931f64ca44248f95222c4b8c039b P bfa6698e37504e508c1ea2f09791f1c8: No bootstrap required, opened a new log
I20260812 09:49:20.162556 11051 ts_tablet_manager.cc:1397] T c107931f64ca44248f95222c4b8c039b P bfa6698e37504e508c1ea2f09791f1c8: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260812 09:49:20.165119 11048 raft_consensus.cc:348] T c107931f64ca44248f95222c4b8c039b P 28952c7c86c640a3877ad743d4d8ac7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28952c7c86c640a3877ad743d4d8ac7f" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35377 } } peers { permanent_uuid: "cb0c589ba2fd456b8ac0377d4d81800b" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 43379 } } peers { permanent_uuid: "bfa6698e37504e508c1ea2f09791f1c8" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 36967 } }
I20260812 09:49:20.165139 11050 raft_consensus.cc:348] T c107931f64ca44248f95222c4b8c039b P cb0c589ba2fd456b8ac0377d4d81800b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28952c7c86c640a3877ad743d4d8ac7f" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35377 } } peers { permanent_uuid: "cb0c589ba2fd456b8ac0377d4d81800b" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 43379 } } peers { permanent_uuid: "bfa6698e37504e508c1ea2f09791f1c8" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 36967 } }
I20260812 09:49:20.165416 11050 raft_consensus.cc:374] T c107931f64ca44248f95222c4b8c039b P cb0c589ba2fd456b8ac0377d4d81800b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:49:20.165518 11050 raft_consensus.cc:729] T c107931f64ca44248f95222c4b8c039b P cb0c589ba2fd456b8ac0377d4d81800b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb0c589ba2fd456b8ac0377d4d81800b, State: Initialized, Role: FOLLOWER
I20260812 09:49:20.165529 11048 raft_consensus.cc:374] T c107931f64ca44248f95222c4b8c039b P 28952c7c86c640a3877ad743d4d8ac7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:49:20.165587 11048 raft_consensus.cc:729] T c107931f64ca44248f95222c4b8c039b P 28952c7c86c640a3877ad743d4d8ac7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28952c7c86c640a3877ad743d4d8ac7f, State: Initialized, Role: FOLLOWER
I20260812 09:49:20.166088 11050 consensus_queue.cc:260] T c107931f64ca44248f95222c4b8c039b P cb0c589ba2fd456b8ac0377d4d81800b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28952c7c86c640a3877ad743d4d8ac7f" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35377 } } peers { permanent_uuid: "cb0c589ba2fd456b8ac0377d4d81800b" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 43379 } } peers { permanent_uuid: "bfa6698e37504e508c1ea2f09791f1c8" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 36967 } }
I20260812 09:49:20.166129 11048 consensus_queue.cc:260] T c107931f64ca44248f95222c4b8c039b P 28952c7c86c640a3877ad743d4d8ac7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28952c7c86c640a3877ad743d4d8ac7f" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35377 } } peers { permanent_uuid: "cb0c589ba2fd456b8ac0377d4d81800b" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 43379 } } peers { permanent_uuid: "bfa6698e37504e508c1ea2f09791f1c8" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 36967 } }
I20260812 09:49:20.166301 11051 raft_consensus.cc:348] T c107931f64ca44248f95222c4b8c039b P bfa6698e37504e508c1ea2f09791f1c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28952c7c86c640a3877ad743d4d8ac7f" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35377 } } peers { permanent_uuid: "cb0c589ba2fd456b8ac0377d4d81800b" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 43379 } } peers { permanent_uuid: "bfa6698e37504e508c1ea2f09791f1c8" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 36967 } }
I20260812 09:49:20.166623 11051 raft_consensus.cc:374] T c107931f64ca44248f95222c4b8c039b P bfa6698e37504e508c1ea2f09791f1c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:49:20.166718 11051 raft_consensus.cc:729] T c107931f64ca44248f95222c4b8c039b P bfa6698e37504e508c1ea2f09791f1c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfa6698e37504e508c1ea2f09791f1c8, State: Initialized, Role: FOLLOWER
I20260812 09:49:20.166740 10577 heartbeater.cc:503] Master 127.13.99.62:39585 was elected leader, sending a full tablet report...
I20260812 09:49:20.167143 11048 ts_tablet_manager.cc:1428] T c107931f64ca44248f95222c4b8c039b P 28952c7c86c640a3877ad743d4d8ac7f: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20260812 09:49:20.167161 11050 ts_tablet_manager.cc:1428] T c107931f64ca44248f95222c4b8c039b P cb0c589ba2fd456b8ac0377d4d81800b: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20260812 09:49:20.167150 11051 consensus_queue.cc:260] T c107931f64ca44248f95222c4b8c039b P bfa6698e37504e508c1ea2f09791f1c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28952c7c86c640a3877ad743d4d8ac7f" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35377 } } peers { permanent_uuid: "cb0c589ba2fd456b8ac0377d4d81800b" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 43379 } } peers { permanent_uuid: "bfa6698e37504e508c1ea2f09791f1c8" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 36967 } }
I20260812 09:49:20.167469 10805 heartbeater.cc:503] Master 127.13.99.62:39585 was elected leader, sending a full tablet report...
I20260812 09:49:20.168017 11021 heartbeater.cc:503] Master 127.13.99.62:39585 was elected leader, sending a full tablet report...
I20260812 09:49:20.168001 11051 ts_tablet_manager.cc:1428] T c107931f64ca44248f95222c4b8c039b P bfa6698e37504e508c1ea2f09791f1c8: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260812 09:49:20.196655 11059 raft_consensus.cc:482] T c107931f64ca44248f95222c4b8c039b P bfa6698e37504e508c1ea2f09791f1c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 09:49:20.196898 11059 raft_consensus.cc:504] T c107931f64ca44248f95222c4b8c039b P bfa6698e37504e508c1ea2f09791f1c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28952c7c86c640a3877ad743d4d8ac7f" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35377 } } peers { permanent_uuid: "cb0c589ba2fd456b8ac0377d4d81800b" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 43379 } } peers { permanent_uuid: "bfa6698e37504e508c1ea2f09791f1c8" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 36967 } }
I20260812 09:49:20.198108 11059 leader_election.cc:288] T c107931f64ca44248f95222c4b8c039b P bfa6698e37504e508c1ea2f09791f1c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28952c7c86c640a3877ad743d4d8ac7f (127.13.99.2:35377), cb0c589ba2fd456b8ac0377d4d81800b (127.13.99.1:43379)
I20260812 09:49:20.202831 11042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:20.198411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.3:34157 (local address 127.13.99.2:35377)
0812 09:49:20.198552 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:20.198556 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:20.198572 (+ 16us) server_negotiation.cc:407] Connection header received
0812 09:49:20.198627 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:20.198630 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:20.198686 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:20.198779 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:49:20.199151 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:20.199928 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:20.201534 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:20.202149 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:20.202255 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:20.202318 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:20.202631 (+ 313us) server_negotiation.cc:299] Negotiation successful
0812 09:49:20.202704 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260812 09:49:20.202831 11045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:20.198607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.3:54307 (local address 127.13.99.1:43379)
0812 09:49:20.198762 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:20.198768 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:20.198864 (+ 96us) server_negotiation.cc:407] Connection header received
0812 09:49:20.198963 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:20.198965 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:20.199005 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:20.199064 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:49:20.199357 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:20.200085 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:20.201534 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:20.202141 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:20.202255 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:20.202318 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:20.202631 (+ 313us) server_negotiation.cc:299] Negotiation successful
0812 09:49:20.202703 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260812 09:49:20.202875 11023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:20.198279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.2:35377 (local address 127.13.99.3:34157)
0812 09:49:20.198393 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:20.198418 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0812 09:49:20.198515 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:49:20.198809 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:49:20.198815 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:49:20.198840 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:49:20.199006 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:49:20.199013 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:49:20.200040 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:49:20.200043 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:49:20.201392 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:49:20.201410 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:49:20.202265 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:49:20.202269 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:49:20.202451 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:20.202519 (+ 68us) client_negotiation.cc:769] Sending connection context
0812 09:49:20.202600 (+ 81us) client_negotiation.cc:240] Negotiation successful
0812 09:49:20.202700 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":101,"spinlock_wait_cycles":38272}
I20260812 09:49:20.202888 11060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:20.198362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.1:43379 (local address 127.13.99.3:54307)
0812 09:49:20.198769 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:20.198782 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 09:49:20.198869 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:49:20.199121 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:49:20.199124 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:49:20.199143 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:49:20.199244 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:49:20.199248 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:49:20.200185 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:49:20.200188 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:49:20.201408 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:49:20.201415 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:49:20.202265 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:49:20.202269 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:49:20.202448 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:20.202519 (+ 71us) client_negotiation.cc:769] Sending connection context
0812 09:49:20.202600 (+ 81us) client_negotiation.cc:240] Negotiation successful
0812 09:49:20.202705 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"mutex_wait_us":104,"thread_start_us":152,"threads_started":1}
I20260812 09:49:20.203557 10720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c107931f64ca44248f95222c4b8c039b" candidate_uuid: "bfa6698e37504e508c1ea2f09791f1c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28952c7c86c640a3877ad743d4d8ac7f" is_pre_election: true
I20260812 09:49:20.203583 10503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c107931f64ca44248f95222c4b8c039b" candidate_uuid: "bfa6698e37504e508c1ea2f09791f1c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb0c589ba2fd456b8ac0377d4d81800b" is_pre_election: true
I20260812 09:49:20.203859 10720 raft_consensus.cc:2454] T c107931f64ca44248f95222c4b8c039b P 28952c7c86c640a3877ad743d4d8ac7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfa6698e37504e508c1ea2f09791f1c8 in term 0.
I20260812 09:49:20.203873 10503 raft_consensus.cc:2454] T c107931f64ca44248f95222c4b8c039b P cb0c589ba2fd456b8ac0377d4d81800b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfa6698e37504e508c1ea2f09791f1c8 in term 0.
I20260812 09:49:20.204370 10837 leader_election.cc:302] T c107931f64ca44248f95222c4b8c039b P bfa6698e37504e508c1ea2f09791f1c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bfa6698e37504e508c1ea2f09791f1c8, cb0c589ba2fd456b8ac0377d4d81800b; no voters:
I20260812 09:49:20.204674 11059 raft_consensus.cc:2781] T c107931f64ca44248f95222c4b8c039b P bfa6698e37504e508c1ea2f09791f1c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 09:49:20.204774 11059 raft_consensus.cc:482] T c107931f64ca44248f95222c4b8c039b P bfa6698e37504e508c1ea2f09791f1c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 09:49:20.204854 11059 raft_consensus.cc:3037] T c107931f64ca44248f95222c4b8c039b P bfa6698e37504e508c1ea2f09791f1c8 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:49:20.205967 11059 raft_consensus.cc:504] T c107931f64ca44248f95222c4b8c039b P bfa6698e37504e508c1ea2f09791f1c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28952c7c86c640a3877ad743d4d8ac7f" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35377 } } peers { permanent_uuid: "cb0c589ba2fd456b8ac0377d4d81800b" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 43379 } } peers { permanent_uuid: "bfa6698e37504e508c1ea2f09791f1c8" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 36967 } }
I20260812 09:49:20.206357 11059 leader_election.cc:288] T c107931f64ca44248f95222c4b8c039b P bfa6698e37504e508c1ea2f09791f1c8 [CANDIDATE]: Term 1 election: Requested vote from peers 28952c7c86c640a3877ad743d4d8ac7f (127.13.99.2:35377), cb0c589ba2fd456b8ac0377d4d81800b (127.13.99.1:43379)
I20260812 09:49:20.206884 10720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c107931f64ca44248f95222c4b8c039b" candidate_uuid: "bfa6698e37504e508c1ea2f09791f1c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28952c7c86c640a3877ad743d4d8ac7f"
I20260812 09:49:20.206883 10503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c107931f64ca44248f95222c4b8c039b" candidate_uuid: "bfa6698e37504e508c1ea2f09791f1c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb0c589ba2fd456b8ac0377d4d81800b"
I20260812 09:49:20.207017 10503 raft_consensus.cc:3037] T c107931f64ca44248f95222c4b8c039b P cb0c589ba2fd456b8ac0377d4d81800b [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:49:20.207017 10720 raft_consensus.cc:3037] T c107931f64ca44248f95222c4b8c039b P 28952c7c86c640a3877ad743d4d8ac7f [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:49:20.208468 10720 raft_consensus.cc:2454] T c107931f64ca44248f95222c4b8c039b P 28952c7c86c640a3877ad743d4d8ac7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfa6698e37504e508c1ea2f09791f1c8 in term 1.
I20260812 09:49:20.208468 10503 raft_consensus.cc:2454] T c107931f64ca44248f95222c4b8c039b P cb0c589ba2fd456b8ac0377d4d81800b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfa6698e37504e508c1ea2f09791f1c8 in term 1.
I20260812 09:49:20.208848 10838 leader_election.cc:302] T c107931f64ca44248f95222c4b8c039b P bfa6698e37504e508c1ea2f09791f1c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28952c7c86c640a3877ad743d4d8ac7f, bfa6698e37504e508c1ea2f09791f1c8; no voters:
I20260812 09:49:20.209048 11059 raft_consensus.cc:2781] T c107931f64ca44248f95222c4b8c039b P bfa6698e37504e508c1ea2f09791f1c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 09:49:20.209369 11059 raft_consensus.cc:686] T c107931f64ca44248f95222c4b8c039b P bfa6698e37504e508c1ea2f09791f1c8 [term 1 LEADER]: Becoming Leader. State: Replica: bfa6698e37504e508c1ea2f09791f1c8, State: Running, Role: LEADER
I20260812 09:49:20.209719 11059 consensus_queue.cc:237] T c107931f64ca44248f95222c4b8c039b P bfa6698e37504e508c1ea2f09791f1c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28952c7c86c640a3877ad743d4d8ac7f" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35377 } } peers { permanent_uuid: "cb0c589ba2fd456b8ac0377d4d81800b" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 43379 } } peers { permanent_uuid: "bfa6698e37504e508c1ea2f09791f1c8" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 36967 } }
I20260812 09:49:20.213119 10264 catalog_manager.cc:5495] T c107931f64ca44248f95222c4b8c039b P bfa6698e37504e508c1ea2f09791f1c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to bfa6698e37504e508c1ea2f09791f1c8 (127.13.99.3). New cstate: current_term: 1 leader_uuid: "bfa6698e37504e508c1ea2f09791f1c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28952c7c86c640a3877ad743d4d8ac7f" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb0c589ba2fd456b8ac0377d4d81800b" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 43379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bfa6698e37504e508c1ea2f09791f1c8" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 36967 } health_report { overall_health: HEALTHY } } }
I20260812 09:49:20.240865 11046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:20.237211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34358 (local address 127.13.99.3:36967)
0812 09:49:20.237364 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:20.237369 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:20.237383 (+ 14us) server_negotiation.cc:407] Connection header received
0812 09:49:20.237542 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:20.237544 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:20.237601 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:20.237714 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 09:49:20.237963 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:20.238472 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:20.239353 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:20.239576 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:20.239684 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:20.240003 (+ 319us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 09:49:20.240423 (+ 420us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 09:49:20.240595 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:20.240638 (+ 43us) server_negotiation.cc:299] Negotiation successful
0812 09:49:20.240696 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260812 09:49:20.247221 10720 raft_consensus.cc:1261] T c107931f64ca44248f95222c4b8c039b P 28952c7c86c640a3877ad743d4d8ac7f [term 1 FOLLOWER]: Refusing update from remote peer bfa6698e37504e508c1ea2f09791f1c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 09:49:20.247221 10503 raft_consensus.cc:1261] T c107931f64ca44248f95222c4b8c039b P cb0c589ba2fd456b8ac0377d4d81800b [term 1 FOLLOWER]: Refusing update from remote peer bfa6698e37504e508c1ea2f09791f1c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 09:49:20.247898 11059 consensus_queue.cc:1035] T c107931f64ca44248f95222c4b8c039b P bfa6698e37504e508c1ea2f09791f1c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb0c589ba2fd456b8ac0377d4d81800b" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 43379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 09:49:20.248111 11063 consensus_queue.cc:1035] T c107931f64ca44248f95222c4b8c039b P bfa6698e37504e508c1ea2f09791f1c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28952c7c86c640a3877ad743d4d8ac7f" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 09:49:20.257714 11067 mvcc.cc:205] Tried to move back new op lower bound from 7317619344365514752 to 7317619344221605888. Current Snapshot: MvccSnapshot[applied={T|T < 7317619344365514752 or (T in {7317619344365514752})}]
I20260812 09:49:20.259131 11070 mvcc.cc:205] Tried to move back new op lower bound from 7317619344365514752 to 7317619344221605888. Current Snapshot: MvccSnapshot[applied={T|T < 7317619344365514752}]
I20260812 09:49:20.264350 10264 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38878:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 09:49:20.264545 10264 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38878:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 09:49:20.266904 10264 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6be03e025f384bafbdaddc22a849f405: Sending DeleteTablet for 3 replicas of tablet c107931f64ca44248f95222c4b8c039b
I20260812 09:49:20.267697 10692 tablet_service.cc:1514] Processing DeleteTablet for tablet c107931f64ca44248f95222c4b8c039b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 09:49:20 UTC) from {username='kudu', principal='kudu/127.13.99.62@KRBTEST.COM'} at 127.0.0.1:33260
I20260812 09:49:20.267771 10919 tablet_service.cc:1514] Processing DeleteTablet for tablet c107931f64ca44248f95222c4b8c039b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 09:49:20 UTC) from {username='kudu', principal='kudu/127.13.99.62@KRBTEST.COM'} at 127.0.0.1:34356
I20260812 09:49:20.268261 11088 tablet_replica.cc:330] T c107931f64ca44248f95222c4b8c039b P bfa6698e37504e508c1ea2f09791f1c8: stopping tablet replica
I20260812 09:49:20.268260 10475 tablet_service.cc:1514] Processing DeleteTablet for tablet c107931f64ca44248f95222c4b8c039b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 09:49:20 UTC) from {username='kudu', principal='kudu/127.13.99.62@KRBTEST.COM'} at 127.0.0.1:55980
I20260812 09:49:20.268643 11088 raft_consensus.cc:2229] T c107931f64ca44248f95222c4b8c039b P bfa6698e37504e508c1ea2f09791f1c8 [term 1 LEADER]: Raft consensus shutting down.
I20260812 09:49:20.268687 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 10363
I20260812 09:49:20.269052 11088 raft_consensus.cc:2258] T c107931f64ca44248f95222c4b8c039b P bfa6698e37504e508c1ea2f09791f1c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 09:49:20.285184 11088 ts_tablet_manager.cc:1905] T c107931f64ca44248f95222c4b8c039b P bfa6698e37504e508c1ea2f09791f1c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 09:49:20.288022 11088 ts_tablet_manager.cc:1918] T c107931f64ca44248f95222c4b8c039b P bfa6698e37504e508c1ea2f09791f1c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 09:49:20.288130 11088 log.cc:1196] T c107931f64ca44248f95222c4b8c039b P bfa6698e37504e508c1ea2f09791f1c8: Deleting WAL directory at /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/minicluster-data/ts-2/wal/wals/c107931f64ca44248f95222c4b8c039b
I20260812 09:49:20.288465 11088 ts_tablet_manager.cc:1939] T c107931f64ca44248f95222c4b8c039b P bfa6698e37504e508c1ea2f09791f1c8: Deleting consensus metadata
I20260812 09:49:20.289824 11087 tablet_replica.cc:330] T c107931f64ca44248f95222c4b8c039b P 28952c7c86c640a3877ad743d4d8ac7f: stopping tablet replica
I20260812 09:49:20.289835 10247 catalog_manager.cc:4841] TS bfa6698e37504e508c1ea2f09791f1c8 (127.13.99.3:36967): tablet c107931f64ca44248f95222c4b8c039b (table test-table [id=5340ffbff22545608ee50b947d6d3385]) successfully deleted
I20260812 09:49:20.290191 11087 raft_consensus.cc:2229] T c107931f64ca44248f95222c4b8c039b P 28952c7c86c640a3877ad743d4d8ac7f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 09:49:20.290529 11087 raft_consensus.cc:2258] T c107931f64ca44248f95222c4b8c039b P 28952c7c86c640a3877ad743d4d8ac7f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 09:49:20.292114 11087 ts_tablet_manager.cc:1905] T c107931f64ca44248f95222c4b8c039b P 28952c7c86c640a3877ad743d4d8ac7f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 09:49:20.294773 11087 ts_tablet_manager.cc:1918] T c107931f64ca44248f95222c4b8c039b P 28952c7c86c640a3877ad743d4d8ac7f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 09:49:20.294871 11087 log.cc:1196] T c107931f64ca44248f95222c4b8c039b P 28952c7c86c640a3877ad743d4d8ac7f: Deleting WAL directory at /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786527905902985-13708-0/minicluster-data/ts-1/wal/wals/c107931f64ca44248f95222c4b8c039b
I20260812 09:49:20.295197 11087 ts_tablet_manager.cc:1939] T c107931f64ca44248f95222c4b8c039b P 28952c7c86c640a3877ad743d4d8ac7f: Deleting consensus metadata
W20260812 09:49:20.296277 10244 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.99.1:43379 (error 108)
I20260812 09:49:20.296684 10245 catalog_manager.cc:4841] TS 28952c7c86c640a3877ad743d4d8ac7f (127.13.99.2:35377): tablet c107931f64ca44248f95222c4b8c039b (table test-table [id=5340ffbff22545608ee50b947d6d3385]) successfully deleted
I20260812 09:49:20.297324 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 10590
I20260812 09:49:20.297540 11040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:20.297205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.1:43379 (local address 127.0.0.1:55994)
0812 09:49:20.297334 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:20.297439 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.1:43379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260812 09:49:20.303282 10244 catalog_manager.cc:4584] TS cb0c589ba2fd456b8ac0377d4d81800b (127.13.99.1:43379): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c107931f64ca44248f95222c4b8c039b: Network error: Client connection negotiation failed: client connection to 127.13.99.1:43379: connect: Connection refused (error 111)
I20260812 09:49:20.307989 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 10811
I20260812 09:49:20.317114 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 10219
2026-08-12T09:49:20Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2584 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-12T09:49:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T09:49:20Z Disabled control of system clock
I20260812 09:49:20.357820 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786527905902985-13708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786527905902985-13708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786527905902985-13708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.62:37287
--webserver_interface=127.13.99.62
--webserver_port=0
--builtin_ntp_servers=127.13.99.20:35561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.99.62:37287
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 09:49:20.469244 11102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:49:20.469488 11102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:49:20.469523 11102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:49:20.473363 11102 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 09:49:20.473408 11102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:49:20.473423 11102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:49:20.473439 11102 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 09:49:20.473454 11102 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 09:49:20.477950 11102 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:35561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786527905902985-13708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786527905902985-13708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.99.62:37287
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.62:37287
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.99.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11102
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786527905902985-13708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:49:20.478873 11102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:49:20.480134 11102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:49:20.488463 11108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:49:20.488512 11107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:49:20.488572 11102 server_base.cc:1034] running on GCE node
W20260812 09:49:20.488512 11112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:49:20.489293 11102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:49:20.490273 11102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:49:20.491460 11102 hybrid_clock.cc:648] HybridClock initialized: now 1786528160491432 us; error 39 us; skew 500 ppm
I20260812 09:49:20.493307 11102 webserver.cc:460] Webserver started at http://127.13.99.62:33007/ using document root <none> and password file <none>
I20260812 09:49:20.493930 11102 fs_manager.cc:359] Metadata directory not provided
I20260812 09:49:20.493996 11102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:49:20.494159 11102 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:49:20.496109 11102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786527905902985-13708-0/minicluster-data/master-0/data/instance:
uuid: "290f1a94d28d437682603b5db4eaf964"
format_stamp: "Formatted at 2026-08-12 09:49:20 on dist-test-slave-jq4t"
server_key: "0157a427bdb545f579e380a0f941c080"
server_key_iv: "25df3a3c05d18a667c78d1a8262c1f2f"
server_key_version: "encryptionkey@0"
I20260812 09:49:20.496690 11102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786527905902985-13708-0/minicluster-data/master-0/wal/instance:
uuid: "290f1a94d28d437682603b5db4eaf964"
format_stamp: "Formatted at 2026-08-12 09:49:20 on dist-test-slave-jq4t"
server_key: "0157a427bdb545f579e380a0f941c080"
server_key_iv: "25df3a3c05d18a667c78d1a8262c1f2f"
server_key_version: "encryptionkey@0"
I20260812 09:49:20.500109 11102 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260812 09:49:20.502441 11122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:49:20.503669 11102 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 09:49:20.503793 11102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786527905902985-13708-0/minicluster-data/master-0/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786527905902985-13708-0/minicluster-data/master-0/wal
uuid: "290f1a94d28d437682603b5db4eaf964"
format_stamp: "Formatted at 2026-08-12 09:49:20 on dist-test-slave-jq4t"
server_key: "0157a427bdb545f579e380a0f941c080"
server_key_iv: "25df3a3c05d18a667c78d1a8262c1f2f"
server_key_version: "encryptionkey@0"
I20260812 09:49:20.503890 11102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786527905902985-13708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786527905902985-13708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786527905902985-13708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:49:20.523537 11102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:49:20.524199 11102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:49:20.524402 11102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:49:20.531101 11102 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.62:37287
I20260812 09:49:20.531113 11213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.62:37287 every 8 connection(s)
I20260812 09:49:20.532037 11102 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
I20260812 09:49:20.533488 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 11102
I20260812 09:49:20.533593 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786527905902985-13708-0/minicluster-data/master-0/wal/instance
I20260812 09:49:20.533921 13708 external_mini_cluster.cc:1428] Setting key 2b7d8e0d979f6fdf53c9aa8ad36beaaa
I20260812 09:49:20.535723 11214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:49:20.541627 11214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 290f1a94d28d437682603b5db4eaf964: Bootstrap starting.
I20260812 09:49:20.543412 11218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:20.535358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42408 (local address 127.13.99.62:37287)
0812 09:49:20.536154 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:20.536170 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:20.536206 (+ 36us) server_negotiation.cc:407] Connection header received
0812 09:49:20.536948 (+ 742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:20.536967 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:20.537349 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:20.537705 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:49:20.538137 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:20.539118 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:20.540017 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:20.540552 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:20.540723 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:20.541659 (+ 936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:49:20.541664 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:49:20.541677 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 09:49:20.542108 (+ 431us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:49:20.542165 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 09:49:20.542188 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:49:20.542301 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:20.542458 (+ 157us) server_negotiation.cc:299] Negotiation successful
0812 09:49:20.542718 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":155,"threads_started":1}
I20260812 09:49:20.544034 11214 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 290f1a94d28d437682603b5db4eaf964: Neither blocks nor log segments found. Creating new log.
I20260812 09:49:20.544945 11214 log.cc:824] T 00000000000000000000000000000000 P 290f1a94d28d437682603b5db4eaf964: Log is configured to *not* fsync() on all Append() calls
I20260812 09:49:20.546911 11214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 290f1a94d28d437682603b5db4eaf964: No bootstrap required, opened a new log
I20260812 09:49:20.550750 11214 raft_consensus.cc:348] T 00000000000000000000000000000000 P 290f1a94d28d437682603b5db4eaf964 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "290f1a94d28d437682603b5db4eaf964" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 37287 } }
I20260812 09:49:20.551070 11214 raft_consensus.cc:374] T 00000000000000000000000000000000 P 290f1a94d28d437682603b5db4eaf964 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:49:20.551143 11214 raft_consensus.cc:729] T 00000000000000000000000000000000 P 290f1a94d28d437682603b5db4eaf964 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 290f1a94d28d437682603b5db4eaf964, State: Initialized, Role: FOLLOWER
I20260812 09:49:20.551677 11214 consensus_queue.cc:260] T 00000000000000000000000000000000 P 290f1a94d28d437682603b5db4eaf964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "290f1a94d28d437682603b5db4eaf964" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 37287 } }
I20260812 09:49:20.551837 11214 raft_consensus.cc:388] T 00000000000000000000000000000000 P 290f1a94d28d437682603b5db4eaf964 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 09:49:20.551903 11214 raft_consensus.cc:482] T 00000000000000000000000000000000 P 290f1a94d28d437682603b5db4eaf964 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 09:49:20.552021 11214 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 290f1a94d28d437682603b5db4eaf964 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:49:20.553432 11214 raft_consensus.cc:504] T 00000000000000000000000000000000 P 290f1a94d28d437682603b5db4eaf964 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "290f1a94d28d437682603b5db4eaf964" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 37287 } }
I20260812 09:49:20.553982 11214 leader_election.cc:302] T 00000000000000000000000000000000 P 290f1a94d28d437682603b5db4eaf964 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 290f1a94d28d437682603b5db4eaf964; no voters:
I20260812 09:49:20.554373 11214 leader_election.cc:288] T 00000000000000000000000000000000 P 290f1a94d28d437682603b5db4eaf964 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 09:49:20.554447 11220 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 290f1a94d28d437682603b5db4eaf964 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 09:49:20.554652 11220 raft_consensus.cc:686] T 00000000000000000000000000000000 P 290f1a94d28d437682603b5db4eaf964 [term 1 LEADER]: Becoming Leader. State: Replica: 290f1a94d28d437682603b5db4eaf964, State: Running, Role: LEADER
I20260812 09:49:20.554946 11220 consensus_queue.cc:237] T 00000000000000000000000000000000 P 290f1a94d28d437682603b5db4eaf964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "290f1a94d28d437682603b5db4eaf964" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 37287 } }
I20260812 09:49:20.555696 11214 sys_catalog.cc:564] T 00000000000000000000000000000000 P 290f1a94d28d437682603b5db4eaf964 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 09:49:20.556137 11220 sys_catalog.cc:455] T 00000000000000000000000000000000 P 290f1a94d28d437682603b5db4eaf964 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 290f1a94d28d437682603b5db4eaf964. Latest consensus state: current_term: 1 leader_uuid: "290f1a94d28d437682603b5db4eaf964" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "290f1a94d28d437682603b5db4eaf964" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 37287 } } }
I20260812 09:49:20.556258 11220 sys_catalog.cc:458] T 00000000000000000000000000000000 P 290f1a94d28d437682603b5db4eaf964 [sys.catalog]: This master's current role is: LEADER
I20260812 09:49:20.556460 11220 sys_catalog.cc:455] T 00000000000000000000000000000000 P 290f1a94d28d437682603b5db4eaf964 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "290f1a94d28d437682603b5db4eaf964" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "290f1a94d28d437682603b5db4eaf964" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 37287 } } }
I20260812 09:49:20.556612 11220 sys_catalog.cc:458] T 00000000000000000000000000000000 P 290f1a94d28d437682603b5db4eaf964 [sys.catalog]: This master's current role is: LEADER
I20260812 09:49:20.556911 11224 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 09:49:20.559661 11224 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 09:49:20.565227 11224 catalog_manager.cc:1347] Generated new cluster ID: b9f26bf842b942e5b47db05a461e5afb
I20260812 09:49:20.565300 11224 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 09:49:20.608675 11224 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 09:49:20.609637 11224 catalog_manager.cc:1504] Loading token signing keys...
I20260812 09:49:20.615283 11224 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 290f1a94d28d437682603b5db4eaf964: Generated new TSK 0
I20260812 09:49:20.615999 11224 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 09:49:20.622524 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786527905902985-13708-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786527905902985-13708-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786527905902985-13708-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786527905902985-13708-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.99.1:0
--local_ip_for_outbound_sockets=127.13.99.1
--webserver_interface=127.13.99.1
--webserver_port=0
--tserver_master_addrs=127.13.99.62:37287
--builtin_ntp_servers=127.13.99.20:35561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 09:49:20.737625 11249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:49:20.737862 11249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:49:20.737900 11249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:49:20.741861 11249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:49:20.741912 11249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:49:20.741964 11249 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.99.1
I20260812 09:49:20.747498 11249 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:35561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786527905902985-13708-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786527905902985-13708-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.99.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786527905902985-13708-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.99.1
--webserver_port=0
--tserver_master_addrs=127.13.99.62:37287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11249
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.99.1
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786527905902985-13708-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:49:20.748589 11249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:49:20.749969 11249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:49:20.759804 11258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:49:20.759863 11256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:49:20.759797 11263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:49:20.760061 11249 server_base.cc:1034] running on GCE node
I20260812 09:49:20.760550 11249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:49:20.761178 11249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:49:20.762374 11249 hybrid_clock.cc:648] HybridClock initialized: now 1786528160762360 us; error 35 us; skew 500 ppm
I20260812 09:49:20.764158 11249 webserver.cc:460] Webserver started at http://127.13.99.1:33689/ using document root <none> and password file <none>
I20260812 09:49:20.764803 11249 fs_manager.cc:359] Metadata directory not provided
I20260812 09:49:20.764901 11249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:49:20.765141 11249 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:49:20.767138 11249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786527905902985-13708-0/minicluster-data/ts-0/data/instance:
uuid: "c11165da1a534f41b7013ff9e19be5e4"
format_stamp: "Formatted at 2026-08-12 09:49:20 on dist-test-slave-jq4t"
server_key: "6d3467988e00469837d6c36010493dd3"
server_key_iv: "fec1817f9ef6a80141be4bd7abc98c80"
server_key_version: "encryptionkey@0"
I20260812 09:49:20.767719 11249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786527905902985-13708-0/minicluster-data/ts-0/wal/instance:
uuid: "c11165da1a534f41b7013ff9e19be5e4"
format_stamp: "Formatted at 2026-08-12 09:49:20 on dist-test-slave-jq4t"
server_key: "6d3467988e00469837d6c36010493dd3"
server_key_iv: "fec1817f9ef6a80141be4bd7abc98c80"
server_key_version: "encryptionkey@0"
I20260812 09:49:20.771301 11249 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260812 09:49:20.773758 11273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:49:20.774928 11249 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260812 09:49:20.775082 11249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786527905902985-13708-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786527905902985-13708-0/minicluster-data/ts-0/wal
uuid: "c11165da1a534f41b7013ff9e19be5e4"
format_stamp: "Formatted at 2026-08-12 09:49:20 on dist-test-slave-jq4t"
server_key: "6d3467988e00469837d6c36010493dd3"
server_key_iv: "fec1817f9ef6a80141be4bd7abc98c80"
server_key_version: "encryptionkey@0"
I20260812 09:49:20.775209 11249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786527905902985-13708-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786527905902985-13708-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786527905902985-13708-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:49:20.797348 11249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:49:20.798000 11249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:49:20.798200 11249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:49:20.798781 11249 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 09:49:20.799717 11249 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:49:20.799789 11249 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 09:49:20.799860 11249 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:49:20.799906 11249 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 09:49:20.808854 11249 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.1:38625
I20260812 09:49:20.808941 11461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.1:38625 every 8 connection(s)
I20260812 09:49:20.809832 11249 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786527905902985-13708-0/minicluster-data/ts-0/data/info.pb
I20260812 09:49:20.819111 11218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:20.811652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.1:59941 (local address 127.13.99.62:37287)
0812 09:49:20.811803 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:20.811807 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:20.812923 (+ 1116us) server_negotiation.cc:407] Connection header received
0812 09:49:20.814358 (+ 1435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:20.814362 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:20.814428 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:20.814537 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:49:20.815418 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:20.815964 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:20.817156 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:20.817382 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:20.817499 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:20.818564 (+ 1065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:49:20.818568 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:49:20.818571 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 09:49:20.818584 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:49:20.818612 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 09:49:20.818621 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:49:20.818701 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:20.818872 (+ 171us) server_negotiation.cc:299] Negotiation successful
0812 09:49:20.818952 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260812 09:49:20.819947 11466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:20.812060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:37287 (local address 127.13.99.1:59941)
0812 09:49:20.812771 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:20.812812 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0812 09:49:20.814137 (+ 1325us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:49:20.814712 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:49:20.814718 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:49:20.814800 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:49:20.815265 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:49:20.815278 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:49:20.816106 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:49:20.816112 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:49:20.817020 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:49:20.817030 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:49:20.817551 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:49:20.817555 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:49:20.817720 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:20.818296 (+ 576us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 09:49:20.818298 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:49:20.818336 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 09:49:20.818398 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:49:20.818703 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:49:20.818716 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:49:20.818725 (+ 9us) client_negotiation.cc:769] Sending connection context
0812 09:49:20.818864 (+ 139us) client_negotiation.cc:240] Negotiation successful
0812 09:49:20.819132 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":123,"threads_started":1}
I20260812 09:49:20.820041 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 11249
I20260812 09:49:20.820174 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786527905902985-13708-0/minicluster-data/ts-0/wal/instance
I20260812 09:49:20.820570 13708 external_mini_cluster.cc:1428] Setting key 471e4db2a42a6cb21dfce94a3a6317f9
I20260812 09:49:20.821332 11462 heartbeater.cc:347] Connected to a master server at 127.13.99.62:37287
I20260812 09:49:20.821588 11462 heartbeater.cc:464] Registering TS with master...
I20260812 09:49:20.822173 11462 heartbeater.cc:511] Master 127.13.99.62:37287 requested a full tablet report, sending...
I20260812 09:49:20.822618 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786527905902985-13708-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786527905902985-13708-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786527905902985-13708-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786527905902985-13708-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.99.2:0
--local_ip_for_outbound_sockets=127.13.99.2
--webserver_interface=127.13.99.2
--webserver_port=0
--tserver_master_addrs=127.13.99.62:37287
--builtin_ntp_servers=127.13.99.20:35561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 09:49:20.823666 11150 ts_manager.cc:194] Registered new tserver with Master: c11165da1a534f41b7013ff9e19be5e4 (127.13.99.1:38625)
I20260812 09:49:20.824858 11150 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.99.1:59941
W20260812 09:49:20.936174 11467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:49:20.936445 11467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:49:20.936522 11467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:49:20.940588 11467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:49:20.940654 11467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:49:20.940740 11467 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.99.2
I20260812 09:49:20.945401 11467 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:35561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786527905902985-13708-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786527905902985-13708-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.99.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786527905902985-13708-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.99.2
--webserver_port=0
--tserver_master_addrs=127.13.99.62:37287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11467
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.99.2
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786527905902985-13708-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:49:20.946394 11467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:49:20.947715 11467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:49:20.957559 11473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:49:20.957573 11477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:49:20.957573 11474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:49:20.957828 11467 server_base.cc:1034] running on GCE node
I20260812 09:49:20.958256 11467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:49:20.958863 11467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:49:20.960050 11467 hybrid_clock.cc:648] HybridClock initialized: now 1786528160960033 us; error 30 us; skew 500 ppm
I20260812 09:49:20.961865 11467 webserver.cc:460] Webserver started at http://127.13.99.2:35875/ using document root <none> and password file <none>
I20260812 09:49:20.962435 11467 fs_manager.cc:359] Metadata directory not provided
I20260812 09:49:20.962527 11467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:49:20.962742 11467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:49:20.964769 11467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786527905902985-13708-0/minicluster-data/ts-1/data/instance:
uuid: "03ee20781cad4822a638ad0bce8c1273"
format_stamp: "Formatted at 2026-08-12 09:49:20 on dist-test-slave-jq4t"
server_key: "cb7ce8719ea7e60da540497c68eb9a09"
server_key_iv: "b3e5d7a2686e0e7b45a4e58c82774e31"
server_key_version: "encryptionkey@0"
I20260812 09:49:20.965312 11467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786527905902985-13708-0/minicluster-data/ts-1/wal/instance:
uuid: "03ee20781cad4822a638ad0bce8c1273"
format_stamp: "Formatted at 2026-08-12 09:49:20 on dist-test-slave-jq4t"
server_key: "cb7ce8719ea7e60da540497c68eb9a09"
server_key_iv: "b3e5d7a2686e0e7b45a4e58c82774e31"
server_key_version: "encryptionkey@0"
I20260812 09:49:20.968926 11467 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 09:49:20.971480 11488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:49:20.972756 11467 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 09:49:20.972904 11467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786527905902985-13708-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786527905902985-13708-0/minicluster-data/ts-1/wal
uuid: "03ee20781cad4822a638ad0bce8c1273"
format_stamp: "Formatted at 2026-08-12 09:49:20 on dist-test-slave-jq4t"
server_key: "cb7ce8719ea7e60da540497c68eb9a09"
server_key_iv: "b3e5d7a2686e0e7b45a4e58c82774e31"
server_key_version: "encryptionkey@0"
I20260812 09:49:20.973016 11467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786527905902985-13708-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786527905902985-13708-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786527905902985-13708-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:49:20.988287 11467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:49:20.989014 11467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:49:20.989215 11467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:49:20.989837 11467 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 09:49:20.990792 11467 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:49:20.990890 11467 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 09:49:20.990962 11467 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:49:20.991008 11467 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 09:49:21.000191 11467 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.2:35783
I20260812 09:49:21.000269 11664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.2:35783 every 8 connection(s)
I20260812 09:49:21.001158 11467 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786527905902985-13708-0/minicluster-data/ts-1/data/info.pb
I20260812 09:49:21.009938 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 11467
I20260812 09:49:21.010087 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786527905902985-13708-0/minicluster-data/ts-1/wal/instance
I20260812 09:49:21.010428 13708 external_mini_cluster.cc:1428] Setting key e156c25bb48dcc278f6a635642c1b023
I20260812 09:49:21.011049 11218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:21.003300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.2:52067 (local address 127.13.99.62:37287)
0812 09:49:21.003453 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:21.003456 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:21.004520 (+ 1064us) server_negotiation.cc:407] Connection header received
0812 09:49:21.005940 (+ 1420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:21.005944 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:21.006007 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:21.006115 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:49:21.007062 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:21.007645 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:21.008945 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:21.009178 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:21.009294 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:21.010508 (+ 1214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:49:21.010511 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:49:21.010513 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 09:49:21.010544 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:49:21.010566 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 09:49:21.010578 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:49:21.010665 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:21.010842 (+ 177us) server_negotiation.cc:299] Negotiation successful
0812 09:49:21.010905 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 09:49:21.012017 11668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:21.003663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:37287 (local address 127.13.99.2:52067)
0812 09:49:21.004336 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:21.004376 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0812 09:49:21.005712 (+ 1336us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:49:21.006289 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:49:21.006296 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:49:21.006378 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:49:21.006898 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:49:21.006920 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:49:21.007781 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:49:21.007784 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:49:21.008812 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:49:21.008823 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:49:21.009336 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:49:21.009341 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:49:21.009508 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:21.010234 (+ 726us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 09:49:21.010236 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:49:21.010274 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 09:49:21.010336 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:49:21.010667 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:49:21.010680 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:49:21.010689 (+ 9us) client_negotiation.cc:769] Sending connection context
0812 09:49:21.010839 (+ 150us) client_negotiation.cc:240] Negotiation successful
0812 09:49:21.011118 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":135,"threads_started":1}
I20260812 09:49:21.012831 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786527905902985-13708-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786527905902985-13708-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786527905902985-13708-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786527905902985-13708-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.99.3:0
--local_ip_for_outbound_sockets=127.13.99.3
--webserver_interface=127.13.99.3
--webserver_port=0
--tserver_master_addrs=127.13.99.62:37287
--builtin_ntp_servers=127.13.99.20:35561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 09:49:21.013276 11665 heartbeater.cc:347] Connected to a master server at 127.13.99.62:37287
I20260812 09:49:21.013535 11665 heartbeater.cc:464] Registering TS with master...
I20260812 09:49:21.014051 11665 heartbeater.cc:511] Master 127.13.99.62:37287 requested a full tablet report, sending...
I20260812 09:49:21.015103 11150 ts_manager.cc:194] Registered new tserver with Master: 03ee20781cad4822a638ad0bce8c1273 (127.13.99.2:35783)
I20260812 09:49:21.015681 11150 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.99.2:52067
W20260812 09:49:21.127813 11669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:49:21.128074 11669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:49:21.128111 11669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:49:21.132113 11669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:49:21.132169 11669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:49:21.132254 11669 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.99.3
I20260812 09:49:21.137009 11669 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:35561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786527905902985-13708-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786527905902985-13708-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.99.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786527905902985-13708-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.99.3
--webserver_port=0
--tserver_master_addrs=127.13.99.62:37287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11669
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.99.3
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786527905902985-13708-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:49:21.138087 11669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:49:21.139497 11669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:49:21.149197 11675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:49:21.149194 11679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:49:21.149309 11669 server_base.cc:1034] running on GCE node
W20260812 09:49:21.149286 11674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:49:21.149941 11669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:49:21.150574 11669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:49:21.151767 11669 hybrid_clock.cc:648] HybridClock initialized: now 1786528161151753 us; error 37 us; skew 500 ppm
I20260812 09:49:21.153721 11669 webserver.cc:460] Webserver started at http://127.13.99.3:45681/ using document root <none> and password file <none>
I20260812 09:49:21.154304 11669 fs_manager.cc:359] Metadata directory not provided
I20260812 09:49:21.154397 11669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:49:21.154640 11669 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:49:21.156844 11669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786527905902985-13708-0/minicluster-data/ts-2/data/instance:
uuid: "6321c4000c6649f0a61c8212d03b4f25"
format_stamp: "Formatted at 2026-08-12 09:49:21 on dist-test-slave-jq4t"
server_key: "b5512afe43af770f1c70c98172ee3849"
server_key_iv: "add007e0891ef1a6ec03c4d118646048"
server_key_version: "encryptionkey@0"
I20260812 09:49:21.157440 11669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786527905902985-13708-0/minicluster-data/ts-2/wal/instance:
uuid: "6321c4000c6649f0a61c8212d03b4f25"
format_stamp: "Formatted at 2026-08-12 09:49:21 on dist-test-slave-jq4t"
server_key: "b5512afe43af770f1c70c98172ee3849"
server_key_iv: "add007e0891ef1a6ec03c4d118646048"
server_key_version: "encryptionkey@0"
I20260812 09:49:21.161073 11669 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 09:49:21.163470 11687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:49:21.164745 11669 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 09:49:21.164875 11669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786527905902985-13708-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786527905902985-13708-0/minicluster-data/ts-2/wal
uuid: "6321c4000c6649f0a61c8212d03b4f25"
format_stamp: "Formatted at 2026-08-12 09:49:21 on dist-test-slave-jq4t"
server_key: "b5512afe43af770f1c70c98172ee3849"
server_key_iv: "add007e0891ef1a6ec03c4d118646048"
server_key_version: "encryptionkey@0"
I20260812 09:49:21.164991 11669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786527905902985-13708-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786527905902985-13708-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786527905902985-13708-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:49:21.206338 11669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:49:21.207137 11669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:49:21.207350 11669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:49:21.208073 11669 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 09:49:21.209139 11669 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:49:21.209213 11669 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 09:49:21.209288 11669 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:49:21.209340 11669 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 09:49:21.218529 11669 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.3:45011
I20260812 09:49:21.218556 11887 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.3:45011 every 8 connection(s)
I20260812 09:49:21.219461 11669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786527905902985-13708-0/minicluster-data/ts-2/data/info.pb
I20260812 09:49:21.221463 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 11669
I20260812 09:49:21.221598 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786527905902985-13708-0/minicluster-data/ts-2/wal/instance
I20260812 09:49:21.221868 13708 external_mini_cluster.cc:1428] Setting key 9f7b00d469855d25365ae3ab58c41263
I20260812 09:49:21.228952 11218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:21.221435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.3:33779 (local address 127.13.99.62:37287)
0812 09:49:21.221629 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:21.221634 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:21.222802 (+ 1168us) server_negotiation.cc:407] Connection header received
0812 09:49:21.224210 (+ 1408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:21.224214 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:21.224269 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:21.224351 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:49:21.225265 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:21.225801 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:21.226973 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:21.227195 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:21.227303 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:21.228419 (+ 1116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:49:21.228423 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:49:21.228426 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 09:49:21.228439 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:49:21.228465 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 09:49:21.228521 (+ 56us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:49:21.228588 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:21.228752 (+ 164us) server_negotiation.cc:299] Negotiation successful
0812 09:49:21.228813 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260812 09:49:21.229797 11894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:21.221869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:37287 (local address 127.13.99.3:33779)
0812 09:49:21.222645 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:21.222699 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0812 09:49:21.224000 (+ 1301us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:49:21.224546 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:49:21.224553 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:49:21.224637 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:49:21.225117 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:49:21.225129 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:49:21.225938 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:49:21.225941 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:49:21.226825 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:49:21.226834 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:49:21.227344 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:49:21.227348 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:49:21.227524 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:21.228159 (+ 635us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 09:49:21.228160 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:49:21.228199 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 09:49:21.228260 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:49:21.228590 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:49:21.228602 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:49:21.228610 (+ 8us) client_negotiation.cc:769] Sending connection context
0812 09:49:21.228749 (+ 139us) client_negotiation.cc:240] Negotiation successful
0812 09:49:21.229020 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":139,"threads_started":1}
I20260812 09:49:21.230911 11889 heartbeater.cc:347] Connected to a master server at 127.13.99.62:37287
I20260812 09:49:21.231168 11889 heartbeater.cc:464] Registering TS with master...
I20260812 09:49:21.231693 11889 heartbeater.cc:511] Master 127.13.99.62:37287 requested a full tablet report, sending...
I20260812 09:49:21.232759 11150 ts_manager.cc:194] Registered new tserver with Master: 6321c4000c6649f0a61c8212d03b4f25 (127.13.99.3:45011)
I20260812 09:49:21.233299 11150 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.99.3:33779
I20260812 09:49:21.235549 13708 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 09:49:21.241092 11218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:21.237561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42414 (local address 127.13.99.62:37287)
0812 09:49:21.237706 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:21.237710 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:21.237825 (+ 115us) server_negotiation.cc:407] Connection header received
0812 09:49:21.237921 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:21.237923 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:21.237962 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:21.238043 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:49:21.238288 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:21.238763 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:21.239584 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:21.239795 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:21.239874 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:21.240669 (+ 795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:49:21.240672 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:49:21.240675 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 09:49:21.240686 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:49:21.240704 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 09:49:21.240712 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:49:21.240771 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:21.240852 (+ 81us) server_negotiation.cc:299] Negotiation successful
0812 09:49:21.240896 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 09:49:21.243762 11150 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42414:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 09:49:21.246798 11150 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 09:49:21.261320 11917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:21.256242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59938 (local address 127.13.99.2:35783)
0812 09:49:21.256986 (+ 744us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:21.256999 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:21.257025 (+ 26us) server_negotiation.cc:407] Connection header received
0812 09:49:21.257099 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:21.257105 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:21.257287 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:21.257442 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:49:21.257734 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:21.258742 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:21.259979 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:21.260642 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:21.260767 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:21.260970 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:21.261077 (+ 107us) server_negotiation.cc:299] Negotiation successful
0812 09:49:21.261171 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"thread_start_us":131,"threads_started":1}
I20260812 09:49:21.261468 11911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:21.255332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.1:38625 (local address 127.0.0.1:33174)
0812 09:49:21.255733 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:21.255787 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0812 09:49:21.255921 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:49:21.256701 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:49:21.256708 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:49:21.256733 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:49:21.256883 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:49:21.256892 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:49:21.258544 (+ 1652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:49:21.258548 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:49:21.259906 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:49:21.259916 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:49:21.261025 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:49:21.261029 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:49:21.261115 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:21.261141 (+ 26us) client_negotiation.cc:769] Sending connection context
0812 09:49:21.261193 (+ 52us) client_negotiation.cc:240] Negotiation successful
0812 09:49:21.261261 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":55,"thread_start_us":99,"threads_started":1}
I20260812 09:49:21.261318 11916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:21.255807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.2:35783 (local address 127.0.0.1:59938)
0812 09:49:21.256416 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:21.256460 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 09:49:21.256590 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:49:21.257420 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:49:21.257425 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:49:21.257455 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:49:21.257613 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:49:21.257620 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:49:21.258856 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:49:21.258859 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:49:21.259835 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:49:21.259845 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:49:21.260769 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:49:21.260773 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:49:21.260879 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:21.260917 (+ 38us) client_negotiation.cc:769] Sending connection context
0812 09:49:21.261037 (+ 120us) client_negotiation.cc:240] Negotiation successful
0812 09:49:21.261156 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"mutex_wait_us":2,"spinlock_wait_cycles":5888,"thread_start_us":84,"threads_started":1}
I20260812 09:49:21.262041 11912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:21.255514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33174 (local address 127.13.99.1:38625)
0812 09:49:21.256041 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:21.256055 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:21.256085 (+ 30us) server_negotiation.cc:407] Connection header received
0812 09:49:21.256168 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:21.256173 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:21.256507 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:21.257089 (+ 582us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:49:21.257227 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:21.258416 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:21.260055 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:21.260897 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:21.261397 (+ 500us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:21.261573 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:21.261627 (+ 54us) server_negotiation.cc:299] Negotiation successful
0812 09:49:21.261720 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":133,"threads_started":1}
I20260812 09:49:21.262531 11913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:21.256083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.3:45011 (local address 127.0.0.1:49632)
0812 09:49:21.257266 (+ 1183us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:21.257279 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 09:49:21.257462 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:49:21.258073 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:49:21.258077 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:49:21.258097 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:49:21.258188 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:49:21.258192 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:49:21.259604 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:49:21.259607 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:49:21.260787 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:49:21.260800 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:49:21.262102 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:49:21.262106 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:49:21.262203 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:21.262230 (+ 27us) client_negotiation.cc:769] Sending connection context
0812 09:49:21.262312 (+ 82us) client_negotiation.cc:240] Negotiation successful
0812 09:49:21.262391 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1118,"mutex_wait_us":4,"spinlock_wait_cycles":5504,"thread_start_us":90,"threads_started":1}
I20260812 09:49:21.262835 11918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:21.256268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49632 (local address 127.13.99.3:45011)
0812 09:49:21.256741 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:21.256753 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:21.257378 (+ 625us) server_negotiation.cc:407] Connection header received
0812 09:49:21.257624 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:21.257630 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:21.257818 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:21.257914 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:49:21.258303 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:21.259467 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:21.260961 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:21.261721 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:21.261821 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:21.261994 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:21.262342 (+ 348us) server_negotiation.cc:299] Negotiation successful
0812 09:49:21.262438 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":140,"threads_started":1}
I20260812 09:49:21.264813 11560 tablet_service.cc:1467] Processing CreateTablet for tablet e79a1eff4f0a4690a2920add072cecc7 (DEFAULT_TABLE table=test-table [id=e7fd2b537e04418c9b5a1fa7981b528a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 09:49:21.264989 11356 tablet_service.cc:1467] Processing CreateTablet for tablet e79a1eff4f0a4690a2920add072cecc7 (DEFAULT_TABLE table=test-table [id=e7fd2b537e04418c9b5a1fa7981b528a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 09:49:21.265416 11766 tablet_service.cc:1467] Processing CreateTablet for tablet e79a1eff4f0a4690a2920add072cecc7 (DEFAULT_TABLE table=test-table [id=e7fd2b537e04418c9b5a1fa7981b528a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 09:49:21.265949 11560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e79a1eff4f0a4690a2920add072cecc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:49:21.266036 11356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e79a1eff4f0a4690a2920add072cecc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:49:21.266337 11766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e79a1eff4f0a4690a2920add072cecc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:49:21.273144 11919 tablet_bootstrap.cc:492] T e79a1eff4f0a4690a2920add072cecc7 P 03ee20781cad4822a638ad0bce8c1273: Bootstrap starting.
I20260812 09:49:21.274151 11920 tablet_bootstrap.cc:492] T e79a1eff4f0a4690a2920add072cecc7 P c11165da1a534f41b7013ff9e19be5e4: Bootstrap starting.
I20260812 09:49:21.275019 11921 tablet_bootstrap.cc:492] T e79a1eff4f0a4690a2920add072cecc7 P 6321c4000c6649f0a61c8212d03b4f25: Bootstrap starting.
I20260812 09:49:21.275512 11919 tablet_bootstrap.cc:654] T e79a1eff4f0a4690a2920add072cecc7 P 03ee20781cad4822a638ad0bce8c1273: Neither blocks nor log segments found. Creating new log.
I20260812 09:49:21.276337 11920 tablet_bootstrap.cc:654] T e79a1eff4f0a4690a2920add072cecc7 P c11165da1a534f41b7013ff9e19be5e4: Neither blocks nor log segments found. Creating new log.
I20260812 09:49:21.276376 11919 log.cc:824] T e79a1eff4f0a4690a2920add072cecc7 P 03ee20781cad4822a638ad0bce8c1273: Log is configured to *not* fsync() on all Append() calls
I20260812 09:49:21.277024 11921 tablet_bootstrap.cc:654] T e79a1eff4f0a4690a2920add072cecc7 P 6321c4000c6649f0a61c8212d03b4f25: Neither blocks nor log segments found. Creating new log.
I20260812 09:49:21.277244 11920 log.cc:824] T e79a1eff4f0a4690a2920add072cecc7 P c11165da1a534f41b7013ff9e19be5e4: Log is configured to *not* fsync() on all Append() calls
I20260812 09:49:21.277796 11921 log.cc:824] T e79a1eff4f0a4690a2920add072cecc7 P 6321c4000c6649f0a61c8212d03b4f25: Log is configured to *not* fsync() on all Append() calls
I20260812 09:49:21.279265 11920 tablet_bootstrap.cc:492] T e79a1eff4f0a4690a2920add072cecc7 P c11165da1a534f41b7013ff9e19be5e4: No bootstrap required, opened a new log
I20260812 09:49:21.279467 11920 ts_tablet_manager.cc:1397] T e79a1eff4f0a4690a2920add072cecc7 P c11165da1a534f41b7013ff9e19be5e4: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260812 09:49:21.279712 11921 tablet_bootstrap.cc:492] T e79a1eff4f0a4690a2920add072cecc7 P 6321c4000c6649f0a61c8212d03b4f25: No bootstrap required, opened a new log
I20260812 09:49:21.279793 11919 tablet_bootstrap.cc:492] T e79a1eff4f0a4690a2920add072cecc7 P 03ee20781cad4822a638ad0bce8c1273: No bootstrap required, opened a new log
I20260812 09:49:21.279917 11921 ts_tablet_manager.cc:1397] T e79a1eff4f0a4690a2920add072cecc7 P 6321c4000c6649f0a61c8212d03b4f25: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260812 09:49:21.279968 11919 ts_tablet_manager.cc:1397] T e79a1eff4f0a4690a2920add072cecc7 P 03ee20781cad4822a638ad0bce8c1273: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260812 09:49:21.282943 11919 raft_consensus.cc:348] T e79a1eff4f0a4690a2920add072cecc7 P 03ee20781cad4822a638ad0bce8c1273 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c11165da1a534f41b7013ff9e19be5e4" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 38625 } } peers { permanent_uuid: "6321c4000c6649f0a61c8212d03b4f25" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 45011 } } peers { permanent_uuid: "03ee20781cad4822a638ad0bce8c1273" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35783 } }
I20260812 09:49:21.283134 11919 raft_consensus.cc:374] T e79a1eff4f0a4690a2920add072cecc7 P 03ee20781cad4822a638ad0bce8c1273 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:49:21.283165 11919 raft_consensus.cc:729] T e79a1eff4f0a4690a2920add072cecc7 P 03ee20781cad4822a638ad0bce8c1273 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03ee20781cad4822a638ad0bce8c1273, State: Initialized, Role: FOLLOWER
I20260812 09:49:21.283596 11919 consensus_queue.cc:260] T e79a1eff4f0a4690a2920add072cecc7 P 03ee20781cad4822a638ad0bce8c1273 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c11165da1a534f41b7013ff9e19be5e4" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 38625 } } peers { permanent_uuid: "6321c4000c6649f0a61c8212d03b4f25" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 45011 } } peers { permanent_uuid: "03ee20781cad4822a638ad0bce8c1273" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35783 } }
I20260812 09:49:21.283814 11920 raft_consensus.cc:348] T e79a1eff4f0a4690a2920add072cecc7 P c11165da1a534f41b7013ff9e19be5e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c11165da1a534f41b7013ff9e19be5e4" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 38625 } } peers { permanent_uuid: "6321c4000c6649f0a61c8212d03b4f25" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 45011 } } peers { permanent_uuid: "03ee20781cad4822a638ad0bce8c1273" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35783 } }
I20260812 09:49:21.283895 11921 raft_consensus.cc:348] T e79a1eff4f0a4690a2920add072cecc7 P 6321c4000c6649f0a61c8212d03b4f25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c11165da1a534f41b7013ff9e19be5e4" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 38625 } } peers { permanent_uuid: "6321c4000c6649f0a61c8212d03b4f25" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 45011 } } peers { permanent_uuid: "03ee20781cad4822a638ad0bce8c1273" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35783 } }
I20260812 09:49:21.284096 11921 raft_consensus.cc:374] T e79a1eff4f0a4690a2920add072cecc7 P 6321c4000c6649f0a61c8212d03b4f25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:49:21.284204 11921 raft_consensus.cc:729] T e79a1eff4f0a4690a2920add072cecc7 P 6321c4000c6649f0a61c8212d03b4f25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6321c4000c6649f0a61c8212d03b4f25, State: Initialized, Role: FOLLOWER
I20260812 09:49:21.284291 11665 heartbeater.cc:503] Master 127.13.99.62:37287 was elected leader, sending a full tablet report...
I20260812 09:49:21.284699 11919 ts_tablet_manager.cc:1428] T e79a1eff4f0a4690a2920add072cecc7 P 03ee20781cad4822a638ad0bce8c1273: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260812 09:49:21.284655 11921 consensus_queue.cc:260] T e79a1eff4f0a4690a2920add072cecc7 P 6321c4000c6649f0a61c8212d03b4f25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c11165da1a534f41b7013ff9e19be5e4" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 38625 } } peers { permanent_uuid: "6321c4000c6649f0a61c8212d03b4f25" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 45011 } } peers { permanent_uuid: "03ee20781cad4822a638ad0bce8c1273" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35783 } }
I20260812 09:49:21.284840 11920 raft_consensus.cc:374] T e79a1eff4f0a4690a2920add072cecc7 P c11165da1a534f41b7013ff9e19be5e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:49:21.284929 11920 raft_consensus.cc:729] T e79a1eff4f0a4690a2920add072cecc7 P c11165da1a534f41b7013ff9e19be5e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c11165da1a534f41b7013ff9e19be5e4, State: Initialized, Role: FOLLOWER
I20260812 09:49:21.285365 11920 consensus_queue.cc:260] T e79a1eff4f0a4690a2920add072cecc7 P c11165da1a534f41b7013ff9e19be5e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c11165da1a534f41b7013ff9e19be5e4" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 38625 } } peers { permanent_uuid: "6321c4000c6649f0a61c8212d03b4f25" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 45011 } } peers { permanent_uuid: "03ee20781cad4822a638ad0bce8c1273" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35783 } }
I20260812 09:49:21.285583 11921 ts_tablet_manager.cc:1428] T e79a1eff4f0a4690a2920add072cecc7 P 6321c4000c6649f0a61c8212d03b4f25: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260812 09:49:21.285964 11889 heartbeater.cc:503] Master 127.13.99.62:37287 was elected leader, sending a full tablet report...
I20260812 09:49:21.286522 11920 ts_tablet_manager.cc:1428] T e79a1eff4f0a4690a2920add072cecc7 P c11165da1a534f41b7013ff9e19be5e4: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260812 09:49:21.286868 11462 heartbeater.cc:503] Master 127.13.99.62:37287 was elected leader, sending a full tablet report...
W20260812 09:49:21.310937 11464 tablet.cc:2369] T e79a1eff4f0a4690a2920add072cecc7 P c11165da1a534f41b7013ff9e19be5e4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 09:49:21.470451 11890 tablet.cc:2369] T e79a1eff4f0a4690a2920add072cecc7 P 6321c4000c6649f0a61c8212d03b4f25: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 09:49:21.490610 11929 raft_consensus.cc:482] T e79a1eff4f0a4690a2920add072cecc7 P 6321c4000c6649f0a61c8212d03b4f25 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 09:49:21.490849 11929 raft_consensus.cc:504] T e79a1eff4f0a4690a2920add072cecc7 P 6321c4000c6649f0a61c8212d03b4f25 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c11165da1a534f41b7013ff9e19be5e4" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 38625 } } peers { permanent_uuid: "6321c4000c6649f0a61c8212d03b4f25" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 45011 } } peers { permanent_uuid: "03ee20781cad4822a638ad0bce8c1273" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35783 } }
I20260812 09:49:21.492025 11929 leader_election.cc:288] T e79a1eff4f0a4690a2920add072cecc7 P 6321c4000c6649f0a61c8212d03b4f25 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c11165da1a534f41b7013ff9e19be5e4 (127.13.99.1:38625), 03ee20781cad4822a638ad0bce8c1273 (127.13.99.2:35783)
I20260812 09:49:21.496592 11894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:21.492280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.2:35783 (local address 127.13.99.3:48583)
0812 09:49:21.492437 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:21.492456 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 09:49:21.492579 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:49:21.492882 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:49:21.492885 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:49:21.492908 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:49:21.493055 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:49:21.493062 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:49:21.494037 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:49:21.494040 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:49:21.495234 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:49:21.495245 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:49:21.496117 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:49:21.496126 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:49:21.496228 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:21.496258 (+ 30us) client_negotiation.cc:769] Sending connection context
0812 09:49:21.496339 (+ 81us) client_negotiation.cc:240] Negotiation successful
0812 09:49:21.496426 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60,"mutex_wait_us":1}
I20260812 09:49:21.496644 11917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:21.492388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.3:48583 (local address 127.13.99.2:35783)
0812 09:49:21.492564 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:21.492570 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:21.492588 (+ 18us) server_negotiation.cc:407] Connection header received
0812 09:49:21.492700 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:21.492707 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:21.492768 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:21.492857 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:49:21.493176 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:21.493929 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:21.495367 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:21.495983 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:21.496091 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:21.496170 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:21.496377 (+ 207us) server_negotiation.cc:299] Negotiation successful
0812 09:49:21.496473 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260812 09:49:21.496706 11912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:21.492507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.3:36919 (local address 127.13.99.1:38625)
0812 09:49:21.492653 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:21.492658 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:21.493047 (+ 389us) server_negotiation.cc:407] Connection header received
0812 09:49:21.493169 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:21.493173 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:21.493226 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:21.493308 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:49:21.493570 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:21.494221 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:21.495528 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:21.496155 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:21.496256 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:21.496299 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:21.496508 (+ 209us) server_negotiation.cc:299] Negotiation successful
0812 09:49:21.496583 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260812 09:49:21.496748 11932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:21.492280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.1:38625 (local address 127.13.99.3:36919)
0812 09:49:21.492945 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:21.492958 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 09:49:21.493055 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:49:21.493323 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:49:21.493325 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:49:21.493339 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:49:21.493441 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:49:21.493446 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:49:21.494327 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:49:21.494330 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:49:21.495414 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:49:21.495421 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:49:21.496262 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:49:21.496265 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:49:21.496372 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:21.496391 (+ 19us) client_negotiation.cc:769] Sending connection context
0812 09:49:21.496458 (+ 67us) client_negotiation.cc:240] Negotiation successful
0812 09:49:21.496566 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"mutex_wait_us":281,"thread_start_us":135,"threads_started":1}
I20260812 09:49:21.497444 11381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e79a1eff4f0a4690a2920add072cecc7" candidate_uuid: "6321c4000c6649f0a61c8212d03b4f25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c11165da1a534f41b7013ff9e19be5e4" is_pre_election: true
I20260812 09:49:21.497443 11587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e79a1eff4f0a4690a2920add072cecc7" candidate_uuid: "6321c4000c6649f0a61c8212d03b4f25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03ee20781cad4822a638ad0bce8c1273" is_pre_election: true
I20260812 09:49:21.497750 11587 raft_consensus.cc:2454] T e79a1eff4f0a4690a2920add072cecc7 P 03ee20781cad4822a638ad0bce8c1273 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6321c4000c6649f0a61c8212d03b4f25 in term 0.
I20260812 09:49:21.497767 11381 raft_consensus.cc:2454] T e79a1eff4f0a4690a2920add072cecc7 P c11165da1a534f41b7013ff9e19be5e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6321c4000c6649f0a61c8212d03b4f25 in term 0.
I20260812 09:49:21.498235 11689 leader_election.cc:302] T e79a1eff4f0a4690a2920add072cecc7 P 6321c4000c6649f0a61c8212d03b4f25 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6321c4000c6649f0a61c8212d03b4f25, c11165da1a534f41b7013ff9e19be5e4; no voters:
I20260812 09:49:21.498525 11929 raft_consensus.cc:2781] T e79a1eff4f0a4690a2920add072cecc7 P 6321c4000c6649f0a61c8212d03b4f25 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 09:49:21.498615 11929 raft_consensus.cc:482] T e79a1eff4f0a4690a2920add072cecc7 P 6321c4000c6649f0a61c8212d03b4f25 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 09:49:21.498658 11929 raft_consensus.cc:3037] T e79a1eff4f0a4690a2920add072cecc7 P 6321c4000c6649f0a61c8212d03b4f25 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:49:21.499692 11929 raft_consensus.cc:504] T e79a1eff4f0a4690a2920add072cecc7 P 6321c4000c6649f0a61c8212d03b4f25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c11165da1a534f41b7013ff9e19be5e4" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 38625 } } peers { permanent_uuid: "6321c4000c6649f0a61c8212d03b4f25" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 45011 } } peers { permanent_uuid: "03ee20781cad4822a638ad0bce8c1273" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35783 } }
I20260812 09:49:21.500051 11929 leader_election.cc:288] T e79a1eff4f0a4690a2920add072cecc7 P 6321c4000c6649f0a61c8212d03b4f25 [CANDIDATE]: Term 1 election: Requested vote from peers c11165da1a534f41b7013ff9e19be5e4 (127.13.99.1:38625), 03ee20781cad4822a638ad0bce8c1273 (127.13.99.2:35783)
I20260812 09:49:21.500337 11381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e79a1eff4f0a4690a2920add072cecc7" candidate_uuid: "6321c4000c6649f0a61c8212d03b4f25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c11165da1a534f41b7013ff9e19be5e4"
I20260812 09:49:21.500474 11381 raft_consensus.cc:3037] T e79a1eff4f0a4690a2920add072cecc7 P c11165da1a534f41b7013ff9e19be5e4 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:49:21.500499 11587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e79a1eff4f0a4690a2920add072cecc7" candidate_uuid: "6321c4000c6649f0a61c8212d03b4f25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03ee20781cad4822a638ad0bce8c1273"
I20260812 09:49:21.500650 11587 raft_consensus.cc:3037] T e79a1eff4f0a4690a2920add072cecc7 P 03ee20781cad4822a638ad0bce8c1273 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:49:21.501621 11381 raft_consensus.cc:2454] T e79a1eff4f0a4690a2920add072cecc7 P c11165da1a534f41b7013ff9e19be5e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6321c4000c6649f0a61c8212d03b4f25 in term 1.
I20260812 09:49:21.501647 11587 raft_consensus.cc:2454] T e79a1eff4f0a4690a2920add072cecc7 P 03ee20781cad4822a638ad0bce8c1273 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6321c4000c6649f0a61c8212d03b4f25 in term 1.
I20260812 09:49:21.501919 11689 leader_election.cc:302] T e79a1eff4f0a4690a2920add072cecc7 P 6321c4000c6649f0a61c8212d03b4f25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6321c4000c6649f0a61c8212d03b4f25, c11165da1a534f41b7013ff9e19be5e4; no voters:
I20260812 09:49:21.502094 11929 raft_consensus.cc:2781] T e79a1eff4f0a4690a2920add072cecc7 P 6321c4000c6649f0a61c8212d03b4f25 [term 1 FOLLOWER]: Leader election won for term 1
W20260812 09:49:21.502316 11667 tablet.cc:2369] T e79a1eff4f0a4690a2920add072cecc7 P 03ee20781cad4822a638ad0bce8c1273: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 09:49:21.502367 11929 raft_consensus.cc:686] T e79a1eff4f0a4690a2920add072cecc7 P 6321c4000c6649f0a61c8212d03b4f25 [term 1 LEADER]: Becoming Leader. State: Replica: 6321c4000c6649f0a61c8212d03b4f25, State: Running, Role: LEADER
I20260812 09:49:21.502701 11929 consensus_queue.cc:237] T e79a1eff4f0a4690a2920add072cecc7 P 6321c4000c6649f0a61c8212d03b4f25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c11165da1a534f41b7013ff9e19be5e4" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 38625 } } peers { permanent_uuid: "6321c4000c6649f0a61c8212d03b4f25" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 45011 } } peers { permanent_uuid: "03ee20781cad4822a638ad0bce8c1273" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35783 } }
I20260812 09:49:21.506163 11148 catalog_manager.cc:5495] T e79a1eff4f0a4690a2920add072cecc7 P 6321c4000c6649f0a61c8212d03b4f25 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6321c4000c6649f0a61c8212d03b4f25 (127.13.99.3). New cstate: current_term: 1 leader_uuid: "6321c4000c6649f0a61c8212d03b4f25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c11165da1a534f41b7013ff9e19be5e4" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 38625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6321c4000c6649f0a61c8212d03b4f25" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 45011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "03ee20781cad4822a638ad0bce8c1273" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20260812 09:49:21.555437 11918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:21.551394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49640 (local address 127.13.99.3:45011)
0812 09:49:21.551558 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:21.551563 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:21.551578 (+ 15us) server_negotiation.cc:407] Connection header received
0812 09:49:21.551691 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:21.551693 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:21.551748 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:21.551862 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 09:49:21.552123 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:21.552702 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:21.553564 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:21.553783 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:21.553896 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:21.554248 (+ 352us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 09:49:21.554909 (+ 661us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 09:49:21.555036 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:21.555210 (+ 174us) server_negotiation.cc:299] Negotiation successful
0812 09:49:21.555270 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260812 09:49:21.561949 11381 raft_consensus.cc:1261] T e79a1eff4f0a4690a2920add072cecc7 P c11165da1a534f41b7013ff9e19be5e4 [term 1 FOLLOWER]: Refusing update from remote peer 6321c4000c6649f0a61c8212d03b4f25: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 09:49:21.561997 11587 raft_consensus.cc:1261] T e79a1eff4f0a4690a2920add072cecc7 P 03ee20781cad4822a638ad0bce8c1273 [term 1 FOLLOWER]: Refusing update from remote peer 6321c4000c6649f0a61c8212d03b4f25: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 09:49:21.562656 11929 consensus_queue.cc:1035] T e79a1eff4f0a4690a2920add072cecc7 P 6321c4000c6649f0a61c8212d03b4f25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03ee20781cad4822a638ad0bce8c1273" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 35783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 09:49:21.562870 11933 consensus_queue.cc:1035] T e79a1eff4f0a4690a2920add072cecc7 P 6321c4000c6649f0a61c8212d03b4f25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c11165da1a534f41b7013ff9e19be5e4" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 38625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 09:49:21.570919 11939 mvcc.cc:205] Tried to move back new op lower bound from 7317619349749915648 to 7317619349518143488. Current Snapshot: MvccSnapshot[applied={T|T < 7317619349749915648}]
I20260812 09:49:21.571928 11941 mvcc.cc:205] Tried to move back new op lower bound from 7317619349749915648 to 7317619349518143488. Current Snapshot: MvccSnapshot[applied={T|T < 7317619349749915648}]
I20260812 09:49:21.578059 11142 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42414:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 09:49:21.578217 11142 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42414:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 09:49:21.580780 11142 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 290f1a94d28d437682603b5db4eaf964: Sending DeleteTablet for 3 replicas of tablet e79a1eff4f0a4690a2920add072cecc7
I20260812 09:49:21.581622 11560 tablet_service.cc:1514] Processing DeleteTablet for tablet e79a1eff4f0a4690a2920add072cecc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 09:49:21 UTC) from {username='slave'} at 127.0.0.1:59938
I20260812 09:49:21.581661 11356 tablet_service.cc:1514] Processing DeleteTablet for tablet e79a1eff4f0a4690a2920add072cecc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 09:49:21 UTC) from {username='slave'} at 127.0.0.1:33174
I20260812 09:49:21.582131 11952 tablet_replica.cc:330] T e79a1eff4f0a4690a2920add072cecc7 P 03ee20781cad4822a638ad0bce8c1273: stopping tablet replica
I20260812 09:49:21.582145 11766 tablet_service.cc:1514] Processing DeleteTablet for tablet e79a1eff4f0a4690a2920add072cecc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 09:49:21 UTC) from {username='slave'} at 127.0.0.1:49632
I20260812 09:49:21.582468 11952 raft_consensus.cc:2229] T e79a1eff4f0a4690a2920add072cecc7 P 03ee20781cad4822a638ad0bce8c1273 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 09:49:21.582607 11955 tablet_replica.cc:330] T e79a1eff4f0a4690a2920add072cecc7 P 6321c4000c6649f0a61c8212d03b4f25: stopping tablet replica
I20260812 09:49:21.582923 11953 tablet_replica.cc:330] T e79a1eff4f0a4690a2920add072cecc7 P c11165da1a534f41b7013ff9e19be5e4: stopping tablet replica
I20260812 09:49:21.582949 11955 raft_consensus.cc:2229] T e79a1eff4f0a4690a2920add072cecc7 P 6321c4000c6649f0a61c8212d03b4f25 [term 1 LEADER]: Raft consensus shutting down.
I20260812 09:49:21.583343 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 11249
I20260812 09:49:21.583329 11955 raft_consensus.cc:2258] T e79a1eff4f0a4690a2920add072cecc7 P 6321c4000c6649f0a61c8212d03b4f25 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 09:49:21.583374 11953 raft_consensus.cc:2229] T e79a1eff4f0a4690a2920add072cecc7 P c11165da1a534f41b7013ff9e19be5e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 09:49:21.589540 11952 raft_consensus.cc:2258] T e79a1eff4f0a4690a2920add072cecc7 P 03ee20781cad4822a638ad0bce8c1273 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 09:49:21.592383 11952 ts_tablet_manager.cc:1905] T e79a1eff4f0a4690a2920add072cecc7 P 03ee20781cad4822a638ad0bce8c1273: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 09:49:21.592391 11955 ts_tablet_manager.cc:1905] T e79a1eff4f0a4690a2920add072cecc7 P 6321c4000c6649f0a61c8212d03b4f25: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 09:49:21.595211 11952 ts_tablet_manager.cc:1918] T e79a1eff4f0a4690a2920add072cecc7 P 03ee20781cad4822a638ad0bce8c1273: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 09:49:21.595377 11952 log.cc:1196] T e79a1eff4f0a4690a2920add072cecc7 P 03ee20781cad4822a638ad0bce8c1273: Deleting WAL directory at /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786527905902985-13708-0/minicluster-data/ts-1/wal/wals/e79a1eff4f0a4690a2920add072cecc7
I20260812 09:49:21.595791 11952 ts_tablet_manager.cc:1939] T e79a1eff4f0a4690a2920add072cecc7 P 03ee20781cad4822a638ad0bce8c1273: Deleting consensus metadata
I20260812 09:49:21.595932 11955 ts_tablet_manager.cc:1918] T e79a1eff4f0a4690a2920add072cecc7 P 6321c4000c6649f0a61c8212d03b4f25: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 09:49:21.596117 11955 log.cc:1196] T e79a1eff4f0a4690a2920add072cecc7 P 6321c4000c6649f0a61c8212d03b4f25: Deleting WAL directory at /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786527905902985-13708-0/minicluster-data/ts-2/wal/wals/e79a1eff4f0a4690a2920add072cecc7
I20260812 09:49:21.596552 11955 ts_tablet_manager.cc:1939] T e79a1eff4f0a4690a2920add072cecc7 P 6321c4000c6649f0a61c8212d03b4f25: Deleting consensus metadata
I20260812 09:49:21.597259 11125 catalog_manager.cc:4841] TS 03ee20781cad4822a638ad0bce8c1273 (127.13.99.2:35783): tablet e79a1eff4f0a4690a2920add072cecc7 (table test-table [id=e7fd2b537e04418c9b5a1fa7981b528a]) successfully deleted
I20260812 09:49:21.597720 11124 catalog_manager.cc:4841] TS 6321c4000c6649f0a61c8212d03b4f25 (127.13.99.3:45011): tablet e79a1eff4f0a4690a2920add072cecc7 (table test-table [id=e7fd2b537e04418c9b5a1fa7981b528a]) successfully deleted
W20260812 09:49:21.598713 11124 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.99.1:38625 (error 108)
I20260812 09:49:21.599628 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 11467
I20260812 09:49:21.599857 11913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:21.599501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.1:38625 (local address 127.0.0.1:33176)
0812 09:49:21.599663 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:21.599749 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.1:38625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260812 09:49:21.607808 11124 catalog_manager.cc:4584] TS c11165da1a534f41b7013ff9e19be5e4 (127.13.99.1:38625): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e79a1eff4f0a4690a2920add072cecc7: Network error: Client connection negotiation failed: client connection to 127.13.99.1:38625: connect: Connection refused (error 111)
I20260812 09:49:21.614787 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 11669
I20260812 09:49:21.630923 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 11102
2026-08-12T09:49:21Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1311 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 09:49:21 dist-test-slave-jq4t krb5kdc[11963](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 09:49:21 dist-test-slave-jq4t krb5kdc[11963](info): set up 2 sockets
Aug 12 09:49:21 dist-test-slave-jq4t krb5kdc[11963](info): commencing operation
krb5kdc: starting...
W20260812 09:49:22.909097 13708 mini_kdc.cc:121] Time spent starting KDC: real 1.250s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/krb5kdc/test-user.keytab.
Aug 12 09:49:22 dist-test-slave-jq4t krb5kdc[11963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T09:49:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T09:49:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.99.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.99.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.99.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab.
Entry for principal kudu/127.13.99.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab.
I20260812 09:49:23.039335 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.62:41933
--webserver_interface=127.13.99.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab
--principal=kudu/127.13.99.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:34207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.99.62:41933
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:49:23.152828 11992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:49:23.153136 11992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:49:23.153179 11992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:49:23.157101 11992 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 09:49:23.157159 11992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:49:23.157179 11992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:49:23.157195 11992 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 09:49:23.157210 11992 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 09:49:23.162411 11992 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:34207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.99.62:41933
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab
--principal=kudu/127.13.99.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.62:41933
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.99.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11992
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:49:23.163368 11992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:49:23.164309 11992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:49:23.173069 12001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:49:23.173069 12004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:49:23.173098 11992 server_base.cc:1034] running on GCE node
W20260812 09:49:23.173121 12002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:49:23.173955 11992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:49:23.175073 11992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:49:23.176271 11992 hybrid_clock.cc:648] HybridClock initialized: now 1786528163176246 us; error 38 us; skew 500 ppm
Aug 12 09:49:23 dist-test-slave-jq4t krb5kdc[11963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528163, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:49:23.182457 11992 init.cc:377] Logged in from keytab as kudu/127.13.99.62@KRBTEST.COM (short username kudu)
I20260812 09:49:23.183395 11992 webserver.cc:460] Webserver started at http://127.13.99.62:45975/ using document root <none> and password file <none>
I20260812 09:49:23.183971 11992 fs_manager.cc:359] Metadata directory not provided
I20260812 09:49:23.184092 11992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:49:23.184340 11992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:49:23.186550 11992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/minicluster-data/master-0/data/instance:
uuid: "a0a5607944dc48b9acf07d32f2f4f32b"
format_stamp: "Formatted at 2026-08-12 09:49:23 on dist-test-slave-jq4t"
server_key: "708cd192a49b2ec755e495da02d890e4"
server_key_iv: "41acc0e1dc1ee48526961af9d3c594a2"
server_key_version: "encryptionkey@0"
I20260812 09:49:23.187136 11992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/minicluster-data/master-0/wal/instance:
uuid: "a0a5607944dc48b9acf07d32f2f4f32b"
format_stamp: "Formatted at 2026-08-12 09:49:23 on dist-test-slave-jq4t"
server_key: "708cd192a49b2ec755e495da02d890e4"
server_key_iv: "41acc0e1dc1ee48526961af9d3c594a2"
server_key_version: "encryptionkey@0"
I20260812 09:49:23.190963 11992 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260812 09:49:23.193518 12014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:49:23.194900 11992 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 09:49:23.195058 11992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/minicluster-data/master-0/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/minicluster-data/master-0/wal
uuid: "a0a5607944dc48b9acf07d32f2f4f32b"
format_stamp: "Formatted at 2026-08-12 09:49:23 on dist-test-slave-jq4t"
server_key: "708cd192a49b2ec755e495da02d890e4"
server_key_iv: "41acc0e1dc1ee48526961af9d3c594a2"
server_key_version: "encryptionkey@0"
I20260812 09:49:23.195178 11992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:49:23.230168 11992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:49:23.233438 11992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:49:23.233664 11992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:49:23.242183 12097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.62:41933 every 8 connection(s)
I20260812 09:49:23.242304 11992 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.62:41933
I20260812 09:49:23.243330 11992 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
I20260812 09:49:23.246330 12098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:49:23.246701 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 11992
I20260812 09:49:23.246855 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/minicluster-data/master-0/wal/instance
I20260812 09:49:23.247154 13708 external_mini_cluster.cc:1428] Setting key 5aa6fbb88eb104ed7fcebff028f2bace
I20260812 09:49:23.253400 12098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0a5607944dc48b9acf07d32f2f4f32b: Bootstrap starting.
I20260812 09:49:23.256206 12098 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a0a5607944dc48b9acf07d32f2f4f32b: Neither blocks nor log segments found. Creating new log.
Aug 12 09:49:23 dist-test-slave-jq4t krb5kdc[11963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
I20260812 09:49:23.257076 12098 log.cc:824] T 00000000000000000000000000000000 P a0a5607944dc48b9acf07d32f2f4f32b: Log is configured to *not* fsync() on all Append() calls
I20260812 09:49:23.259488 12098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0a5607944dc48b9acf07d32f2f4f32b: No bootstrap required, opened a new log
I20260812 09:49:23.262590 12102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:23.248583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50206 (local address 127.13.99.62:41933)
0812 09:49:23.249024 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:23.249048 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:23.249084 (+ 36us) server_negotiation.cc:407] Connection header received
0812 09:49:23.249742 (+ 658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:23.249759 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:23.250081 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:23.250363 (+ 282us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:49:23.251345 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:23.252362 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:23.253558 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:23.254062 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:23.254194 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:23.257145 (+ 2951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:49:23.257153 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:49:23.257170 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:49:23.257204 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:49:23.259259 (+ 2055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:49:23.259748 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:49:23.259754 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:49:23.259760 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:49:23.259910 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:49:23.260179 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:49:23.260182 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:49:23.260184 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:49:23.260716 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:49:23.260920 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:23.261227 (+ 307us) server_negotiation.cc:299] Negotiation successful
0812 09:49:23.261539 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":164,"threads_started":1}
I20260812 09:49:23.264201 12098 raft_consensus.cc:348] T 00000000000000000000000000000000 P a0a5607944dc48b9acf07d32f2f4f32b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0a5607944dc48b9acf07d32f2f4f32b" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 41933 } }
I20260812 09:49:23.264415 12098 raft_consensus.cc:374] T 00000000000000000000000000000000 P a0a5607944dc48b9acf07d32f2f4f32b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:49:23.264484 12098 raft_consensus.cc:729] T 00000000000000000000000000000000 P a0a5607944dc48b9acf07d32f2f4f32b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0a5607944dc48b9acf07d32f2f4f32b, State: Initialized, Role: FOLLOWER
I20260812 09:49:23.265079 12098 consensus_queue.cc:260] T 00000000000000000000000000000000 P a0a5607944dc48b9acf07d32f2f4f32b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0a5607944dc48b9acf07d32f2f4f32b" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 41933 } }
I20260812 09:49:23.265211 12098 raft_consensus.cc:388] T 00000000000000000000000000000000 P a0a5607944dc48b9acf07d32f2f4f32b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 09:49:23.265296 12098 raft_consensus.cc:482] T 00000000000000000000000000000000 P a0a5607944dc48b9acf07d32f2f4f32b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 09:49:23.265398 12098 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a0a5607944dc48b9acf07d32f2f4f32b [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:49:23.266315 12098 raft_consensus.cc:504] T 00000000000000000000000000000000 P a0a5607944dc48b9acf07d32f2f4f32b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0a5607944dc48b9acf07d32f2f4f32b" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 41933 } }
I20260812 09:49:23.266659 12098 leader_election.cc:302] T 00000000000000000000000000000000 P a0a5607944dc48b9acf07d32f2f4f32b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a0a5607944dc48b9acf07d32f2f4f32b; no voters:
I20260812 09:49:23.267040 12098 leader_election.cc:288] T 00000000000000000000000000000000 P a0a5607944dc48b9acf07d32f2f4f32b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 09:49:23.267170 12104 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a0a5607944dc48b9acf07d32f2f4f32b [term 1 FOLLOWER]: Leader election won for term 1
I20260812 09:49:23.267433 12104 raft_consensus.cc:686] T 00000000000000000000000000000000 P a0a5607944dc48b9acf07d32f2f4f32b [term 1 LEADER]: Becoming Leader. State: Replica: a0a5607944dc48b9acf07d32f2f4f32b, State: Running, Role: LEADER
I20260812 09:49:23.267781 12104 consensus_queue.cc:237] T 00000000000000000000000000000000 P a0a5607944dc48b9acf07d32f2f4f32b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0a5607944dc48b9acf07d32f2f4f32b" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 41933 } }
I20260812 09:49:23.268314 12098 sys_catalog.cc:564] T 00000000000000000000000000000000 P a0a5607944dc48b9acf07d32f2f4f32b [sys.catalog]: configured and running, proceeding with master startup.
I20260812 09:49:23.269578 12106 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0a5607944dc48b9acf07d32f2f4f32b [sys.catalog]: SysCatalogTable state changed. Reason: New leader a0a5607944dc48b9acf07d32f2f4f32b. Latest consensus state: current_term: 1 leader_uuid: "a0a5607944dc48b9acf07d32f2f4f32b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0a5607944dc48b9acf07d32f2f4f32b" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 41933 } } }
I20260812 09:49:23.269578 12105 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0a5607944dc48b9acf07d32f2f4f32b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a0a5607944dc48b9acf07d32f2f4f32b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0a5607944dc48b9acf07d32f2f4f32b" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 41933 } } }
I20260812 09:49:23.269737 12106 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0a5607944dc48b9acf07d32f2f4f32b [sys.catalog]: This master's current role is: LEADER
I20260812 09:49:23.269737 12105 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0a5607944dc48b9acf07d32f2f4f32b [sys.catalog]: This master's current role is: LEADER
I20260812 09:49:23.270790 12117 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 09:49:23.273568 12117 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 09:49:23.278874 12117 catalog_manager.cc:1347] Generated new cluster ID: 9a89d69486544e32ab25158ff382a800
I20260812 09:49:23.278939 12117 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 09:49:23.295861 12117 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 09:49:23.297353 12117 catalog_manager.cc:1504] Loading token signing keys...
I20260812 09:49:23.302533 12117 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a0a5607944dc48b9acf07d32f2f4f32b: Generated new TSK 0
I20260812 09:49:23.303192 12117 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.99.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.99.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.99.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.1.keytab.
Entry for principal kudu/127.13.99.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.1.keytab.
I20260812 09:49:23.336375 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.99.1:0
--local_ip_for_outbound_sockets=127.13.99.1
--webserver_interface=127.13.99.1
--webserver_port=0
--tserver_master_addrs=127.13.99.62:41933
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.1.keytab
--principal=kudu/127.13.99.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:34207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:49:23.454227 12140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:49:23.454478 12140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:49:23.454521 12140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:49:23.458606 12140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:49:23.458664 12140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:49:23.458721 12140 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.99.1
I20260812 09:49:23.464015 12140 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:34207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.1.keytab
--principal=kudu/127.13.99.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.99.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.99.1
--webserver_port=0
--tserver_master_addrs=127.13.99.62:41933
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12140
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.99.1
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:49:23.465077 12140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:49:23.466084 12140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:49:23.475811 12154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:49:23.475826 12152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:49:23.475811 12147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:49:23.476091 12140 server_base.cc:1034] running on GCE node
I20260812 09:49:23.476619 12140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:49:23.477253 12140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:49:23.478420 12140 hybrid_clock.cc:648] HybridClock initialized: now 1786528163478405 us; error 29 us; skew 500 ppm
Aug 12 09:49:23 dist-test-slave-jq4t krb5kdc[11963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528163, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:49:23.481261 12140 init.cc:377] Logged in from keytab as kudu/127.13.99.1@KRBTEST.COM (short username kudu)
I20260812 09:49:23.482163 12140 webserver.cc:460] Webserver started at http://127.13.99.1:45969/ using document root <none> and password file <none>
I20260812 09:49:23.482725 12140 fs_manager.cc:359] Metadata directory not provided
I20260812 09:49:23.482787 12140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:49:23.483034 12140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:49:23.485213 12140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/minicluster-data/ts-0/data/instance:
uuid: "7a208411fc564d5591d4bf455ae2f0bf"
format_stamp: "Formatted at 2026-08-12 09:49:23 on dist-test-slave-jq4t"
server_key: "ac7bace73d7f623344c7c1b40084f397"
server_key_iv: "35c717ab36814f095929a32ffb42a569"
server_key_version: "encryptionkey@0"
I20260812 09:49:23.485895 12140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/minicluster-data/ts-0/wal/instance:
uuid: "7a208411fc564d5591d4bf455ae2f0bf"
format_stamp: "Formatted at 2026-08-12 09:49:23 on dist-test-slave-jq4t"
server_key: "ac7bace73d7f623344c7c1b40084f397"
server_key_iv: "35c717ab36814f095929a32ffb42a569"
server_key_version: "encryptionkey@0"
I20260812 09:49:23.489508 12140 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260812 09:49:23.491791 12163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:49:23.493043 12140 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260812 09:49:23.493176 12140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/minicluster-data/ts-0/wal
uuid: "7a208411fc564d5591d4bf455ae2f0bf"
format_stamp: "Formatted at 2026-08-12 09:49:23 on dist-test-slave-jq4t"
server_key: "ac7bace73d7f623344c7c1b40084f397"
server_key_iv: "35c717ab36814f095929a32ffb42a569"
server_key_version: "encryptionkey@0"
I20260812 09:49:23.493283 12140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:49:23.503966 12140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:49:23.506597 12140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:49:23.506793 12140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:49:23.507345 12140 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 09:49:23.508160 12140 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:49:23.508234 12140 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 09:49:23.508302 12140 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:49:23.508355 12140 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 09:49:23.517474 12140 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.1:38513
I20260812 09:49:23.517488 12332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.1:38513 every 8 connection(s)
I20260812 09:49:23.518539 12140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/minicluster-data/ts-0/data/info.pb
I20260812 09:49:23.523892 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 12140
I20260812 09:49:23.524047 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/minicluster-data/ts-0/wal/instance
I20260812 09:49:23.524348 13708 external_mini_cluster.cc:1428] Setting key 865186cd175548196eedeb9e2aaed9bd
Aug 12 09:49:23 dist-test-slave-jq4t krb5kdc[11963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528163, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.1@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
I20260812 09:49:23.533260 12102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:23.520363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.1:38457 (local address 127.13.99.62:41933)
0812 09:49:23.520537 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:23.520542 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:23.521413 (+ 871us) server_negotiation.cc:407] Connection header received
0812 09:49:23.522690 (+ 1277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:23.522693 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:23.522740 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:23.522815 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:49:23.524126 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:23.524853 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:23.526404 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:23.526643 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:23.526733 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:23.529516 (+ 2783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:49:23.529519 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:49:23.529521 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:49:23.529545 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:49:23.530972 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:49:23.531501 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:49:23.531509 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:49:23.531511 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:49:23.531589 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:49:23.532023 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:49:23.532026 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:49:23.532027 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:49:23.532213 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:49:23.532297 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:23.533027 (+ 730us) server_negotiation.cc:299] Negotiation successful
0812 09:49:23.533140 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260812 09:49:23.534579 12336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:23.520821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:41933 (local address 127.13.99.1:38457)
0812 09:49:23.521266 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:23.521307 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0812 09:49:23.522498 (+ 1191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:49:23.522958 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:49:23.522964 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:49:23.523310 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:49:23.523884 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:49:23.523907 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:49:23.525009 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:49:23.525014 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:49:23.526247 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:49:23.526262 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:49:23.526808 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:49:23.526815 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:49:23.527063 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:23.527732 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:49:23.527758 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:49:23.529330 (+ 1572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:49:23.531099 (+ 1769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:49:23.531105 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:49:23.531120 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:49:23.531381 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:49:23.531690 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:49:23.531696 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:49:23.531701 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:49:23.531894 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:49:23.532336 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:49:23.532345 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:49:23.532742 (+ 397us) client_negotiation.cc:769] Sending connection context
0812 09:49:23.533004 (+ 262us) client_negotiation.cc:240] Negotiation successful
0812 09:49:23.533476 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":123,"threads_started":1}
WARNING: no policy specified for kudu/127.13.99.2@KRBTEST.COM; defaulting to no policy
I20260812 09:49:23.535967 12333 heartbeater.cc:347] Connected to a master server at 127.13.99.62:41933
I20260812 09:49:23.536233 12333 heartbeater.cc:464] Registering TS with master...
I20260812 09:49:23.536784 12333 heartbeater.cc:511] Master 127.13.99.62:41933 requested a full tablet report, sending...
I20260812 09:49:23.538429 12039 ts_manager.cc:194] Registered new tserver with Master: 7a208411fc564d5591d4bf455ae2f0bf (127.13.99.1:38513)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.99.2@KRBTEST.COM" created.
I20260812 09:49:23.539628 12039 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.99.1@KRBTEST.COM'} at 127.13.99.1:38457
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.99.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.2.keytab.
Entry for principal kudu/127.13.99.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.2.keytab.
I20260812 09:49:23.555724 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.99.2:0
--local_ip_for_outbound_sockets=127.13.99.2
--webserver_interface=127.13.99.2
--webserver_port=0
--tserver_master_addrs=127.13.99.62:41933
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.2.keytab
--principal=kudu/127.13.99.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:34207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:49:23.670940 12343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:49:23.671197 12343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:49:23.671262 12343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:49:23.675318 12343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:49:23.675400 12343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:49:23.675480 12343 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.99.2
I20260812 09:49:23.680585 12343 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:34207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.2.keytab
--principal=kudu/127.13.99.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.99.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.99.2
--webserver_port=0
--tserver_master_addrs=127.13.99.62:41933
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12343
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.99.2
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:49:23.681530 12343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:49:23.682539 12343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:49:23.692255 12354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:49:23.692270 12356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:49:23.692256 12352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:49:23.692577 12343 server_base.cc:1034] running on GCE node
I20260812 09:49:23.693024 12343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:49:23.693642 12343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:49:23.694849 12343 hybrid_clock.cc:648] HybridClock initialized: now 1786528163694826 us; error 30 us; skew 500 ppm
Aug 12 09:49:23 dist-test-slave-jq4t krb5kdc[11963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528163, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:49:23.697795 12343 init.cc:377] Logged in from keytab as kudu/127.13.99.2@KRBTEST.COM (short username kudu)
I20260812 09:49:23.698683 12343 webserver.cc:460] Webserver started at http://127.13.99.2:34889/ using document root <none> and password file <none>
I20260812 09:49:23.699343 12343 fs_manager.cc:359] Metadata directory not provided
I20260812 09:49:23.699440 12343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:49:23.699678 12343 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:49:23.701862 12343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/minicluster-data/ts-1/data/instance:
uuid: "3c014875deec4513bbd5d4b1739cd4ae"
format_stamp: "Formatted at 2026-08-12 09:49:23 on dist-test-slave-jq4t"
server_key: "ca69f5a165ba2093f5165af94f3815d5"
server_key_iv: "e6f630afb1c3cf0043a8b8757ceca3e5"
server_key_version: "encryptionkey@0"
I20260812 09:49:23.702467 12343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/minicluster-data/ts-1/wal/instance:
uuid: "3c014875deec4513bbd5d4b1739cd4ae"
format_stamp: "Formatted at 2026-08-12 09:49:23 on dist-test-slave-jq4t"
server_key: "ca69f5a165ba2093f5165af94f3815d5"
server_key_iv: "e6f630afb1c3cf0043a8b8757ceca3e5"
server_key_version: "encryptionkey@0"
I20260812 09:49:23.706190 12343 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260812 09:49:23.708632 12367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:49:23.709795 12343 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260812 09:49:23.709955 12343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/minicluster-data/ts-1/wal
uuid: "3c014875deec4513bbd5d4b1739cd4ae"
format_stamp: "Formatted at 2026-08-12 09:49:23 on dist-test-slave-jq4t"
server_key: "ca69f5a165ba2093f5165af94f3815d5"
server_key_iv: "e6f630afb1c3cf0043a8b8757ceca3e5"
server_key_version: "encryptionkey@0"
I20260812 09:49:23.710064 12343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:49:23.722163 12343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:49:23.724995 12343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:49:23.725193 12343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:49:23.725744 12343 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 09:49:23.726543 12343 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:49:23.726614 12343 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 09:49:23.726683 12343 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:49:23.726737 12343 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 09:49:23.735734 12343 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.2:33917
I20260812 09:49:23.735755 12552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.2:33917 every 8 connection(s)
I20260812 09:49:23.736647 12343 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/minicluster-data/ts-1/data/info.pb
I20260812 09:49:23.743001 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 12343
I20260812 09:49:23.743151 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/minicluster-data/ts-1/wal/instance
I20260812 09:49:23.743434 13708 external_mini_cluster.cc:1428] Setting key e043df8b4f900ab9df3c70d365123fff
Aug 12 09:49:23 dist-test-slave-jq4t krb5kdc[11963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528163, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.2@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
I20260812 09:49:23.750852 12102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:23.738515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.2:36485 (local address 127.13.99.62:41933)
0812 09:49:23.738653 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:23.738657 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:23.739563 (+ 906us) server_negotiation.cc:407] Connection header received
0812 09:49:23.740928 (+ 1365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:23.740932 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:23.740991 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:23.741074 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:49:23.742233 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:23.742807 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:23.744117 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:23.744406 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:23.744541 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:23.747150 (+ 2609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:49:23.747154 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:49:23.747157 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:49:23.747192 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:49:23.748713 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:49:23.749246 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:49:23.749250 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:49:23.749252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:49:23.749345 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:49:23.749690 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:49:23.749694 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:49:23.749695 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:49:23.749887 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:49:23.749996 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:23.750544 (+ 548us) server_negotiation.cc:299] Negotiation successful
0812 09:49:23.750669 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260812 09:49:23.751660 12558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:23.738902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:41933 (local address 127.13.99.2:36485)
0812 09:49:23.739397 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:23.739440 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0812 09:49:23.740699 (+ 1259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:49:23.741247 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:49:23.741254 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:49:23.741606 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:49:23.742082 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:49:23.742096 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:49:23.742953 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:49:23.742956 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:49:23.743987 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:49:23.743998 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:49:23.744602 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:49:23.744606 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:49:23.744822 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:23.745496 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:49:23.745513 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:49:23.746969 (+ 1456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:49:23.748835 (+ 1866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:49:23.748841 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:49:23.748856 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:49:23.749133 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:49:23.749442 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:49:23.749446 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:49:23.749448 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:49:23.749587 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:49:23.749996 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:49:23.750003 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:49:23.750306 (+ 303us) client_negotiation.cc:769] Sending connection context
0812 09:49:23.750497 (+ 191us) client_negotiation.cc:240] Negotiation successful
0812 09:49:23.750829 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":128,"threads_started":1}
I20260812 09:49:23.752867 12554 heartbeater.cc:347] Connected to a master server at 127.13.99.62:41933
I20260812 09:49:23.753340 12554 heartbeater.cc:464] Registering TS with master...
I20260812 09:49:23.753858 12554 heartbeater.cc:511] Master 127.13.99.62:41933 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.13.99.3@KRBTEST.COM; defaulting to no policy
I20260812 09:49:23.754957 12039 ts_manager.cc:194] Registered new tserver with Master: 3c014875deec4513bbd5d4b1739cd4ae (127.13.99.2:33917)
I20260812 09:49:23.755591 12039 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.99.2@KRBTEST.COM'} at 127.13.99.2:36485
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.99.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.99.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.3.keytab.
Entry for principal kudu/127.13.99.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.3.keytab.
I20260812 09:49:23.773571 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.99.3:0
--local_ip_for_outbound_sockets=127.13.99.3
--webserver_interface=127.13.99.3
--webserver_port=0
--tserver_master_addrs=127.13.99.62:41933
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.3.keytab
--principal=kudu/127.13.99.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:34207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:49:23.886337 12562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:49:23.886579 12562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:49:23.886646 12562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:49:23.890695 12562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:49:23.890748 12562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:49:23.890856 12562 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.99.3
I20260812 09:49:23.896021 12562 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:34207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.3.keytab
--principal=kudu/127.13.99.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.99.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.99.3
--webserver_port=0
--tserver_master_addrs=127.13.99.62:41933
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12562
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.99.3
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:49:23.897096 12562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:49:23.898190 12562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:49:23.908080 12575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:49:23.908102 12572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:49:23.908080 12573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:49:23.908231 12562 server_base.cc:1034] running on GCE node
I20260812 09:49:23.908775 12562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:49:23.909417 12562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:49:23.910599 12562 hybrid_clock.cc:648] HybridClock initialized: now 1786528163910585 us; error 35 us; skew 500 ppm
Aug 12 09:49:23 dist-test-slave-jq4t krb5kdc[11963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528163, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:49:23.913458 12562 init.cc:377] Logged in from keytab as kudu/127.13.99.3@KRBTEST.COM (short username kudu)
I20260812 09:49:23.914391 12562 webserver.cc:460] Webserver started at http://127.13.99.3:39393/ using document root <none> and password file <none>
I20260812 09:49:23.914970 12562 fs_manager.cc:359] Metadata directory not provided
I20260812 09:49:23.915067 12562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:49:23.915307 12562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:49:23.917457 12562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/minicluster-data/ts-2/data/instance:
uuid: "d264c1416cbd4e3aaf5af1d78159e779"
format_stamp: "Formatted at 2026-08-12 09:49:23 on dist-test-slave-jq4t"
server_key: "c96cf7b8d0cac22fdc0a1d49b7ebda50"
server_key_iv: "8f46b674b2d41724b119480a607e4eab"
server_key_version: "encryptionkey@0"
I20260812 09:49:23.918030 12562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/minicluster-data/ts-2/wal/instance:
uuid: "d264c1416cbd4e3aaf5af1d78159e779"
format_stamp: "Formatted at 2026-08-12 09:49:23 on dist-test-slave-jq4t"
server_key: "c96cf7b8d0cac22fdc0a1d49b7ebda50"
server_key_iv: "8f46b674b2d41724b119480a607e4eab"
server_key_version: "encryptionkey@0"
I20260812 09:49:23.921705 12562 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 09:49:23.924072 12589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:49:23.925282 12562 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 09:49:23.925417 12562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/minicluster-data/ts-2/wal
uuid: "d264c1416cbd4e3aaf5af1d78159e779"
format_stamp: "Formatted at 2026-08-12 09:49:23 on dist-test-slave-jq4t"
server_key: "c96cf7b8d0cac22fdc0a1d49b7ebda50"
server_key_iv: "8f46b674b2d41724b119480a607e4eab"
server_key_version: "encryptionkey@0"
I20260812 09:49:23.925525 12562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:49:23.939040 12562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:49:23.942118 12562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:49:23.942320 12562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:49:23.942904 12562 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 09:49:23.943730 12562 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:49:23.943804 12562 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 09:49:23.943897 12562 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:49:23.943951 12562 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 09:49:23.953109 12562 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.3:46315
I20260812 09:49:23.953127 12775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.3:46315 every 8 connection(s)
I20260812 09:49:23.954077 12562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/minicluster-data/ts-2/data/info.pb
I20260812 09:49:23.960289 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 12562
I20260812 09:49:23.960395 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/minicluster-data/ts-2/wal/instance
I20260812 09:49:23.960701 13708 external_mini_cluster.cc:1428] Setting key e346dd92fae0e805f62037639dc1f07a
Aug 12 09:49:23 dist-test-slave-jq4t krb5kdc[11963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528163, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.3@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
I20260812 09:49:23.967821 12102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:23.955861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.3:47177 (local address 127.13.99.62:41933)
0812 09:49:23.956022 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:23.956025 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:23.956874 (+ 849us) server_negotiation.cc:407] Connection header received
0812 09:49:23.958173 (+ 1299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:23.958176 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:23.958237 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:23.958341 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:49:23.959472 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:23.960045 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:23.961538 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:23.961764 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:23.961914 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:23.964385 (+ 2471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:49:23.964394 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:49:23.964396 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:49:23.964424 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:49:23.965733 (+ 1309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:49:23.966254 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:49:23.966258 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:49:23.966261 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:49:23.966350 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:49:23.966727 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:49:23.966730 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:49:23.966732 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:49:23.966930 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:49:23.967044 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:23.967530 (+ 486us) server_negotiation.cc:299] Negotiation successful
0812 09:49:23.967653 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260812 09:49:23.968690 12780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:23.956232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:41933 (local address 127.13.99.3:47177)
0812 09:49:23.956733 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:23.956774 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0812 09:49:23.957962 (+ 1188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:49:23.958518 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:49:23.958525 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:49:23.958863 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:49:23.959322 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:49:23.959335 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:49:23.960204 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:49:23.960207 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:49:23.961398 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:49:23.961410 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:49:23.961923 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:49:23.961927 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:49:23.962111 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:23.962747 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:49:23.962766 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:49:23.964214 (+ 1448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:49:23.965855 (+ 1641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:49:23.965862 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:49:23.965877 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:49:23.966141 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:49:23.966474 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:49:23.966477 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:49:23.966479 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:49:23.966621 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:49:23.967043 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:49:23.967049 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:49:23.967286 (+ 237us) client_negotiation.cc:769] Sending connection context
0812 09:49:23.967483 (+ 197us) client_negotiation.cc:240] Negotiation successful
0812 09:49:23.967793 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":129,"threads_started":1}
I20260812 09:49:23.969758 12776 heartbeater.cc:347] Connected to a master server at 127.13.99.62:41933
I20260812 09:49:23.970019 12776 heartbeater.cc:464] Registering TS with master...
I20260812 09:49:23.970546 12776 heartbeater.cc:511] Master 127.13.99.62:41933 requested a full tablet report, sending...
I20260812 09:49:23.971570 12039 ts_manager.cc:194] Registered new tserver with Master: d264c1416cbd4e3aaf5af1d78159e779 (127.13.99.3:46315)
I20260812 09:49:23.972149 12039 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.99.3@KRBTEST.COM'} at 127.13.99.3:47177
I20260812 09:49:23.974573 13708 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 09:49:23.983911 12102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:23.976540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50212 (local address 127.13.99.62:41933)
0812 09:49:23.976685 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:23.976690 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:23.976901 (+ 211us) server_negotiation.cc:407] Connection header received
0812 09:49:23.977002 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:23.977004 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:23.977042 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:23.977106 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:49:23.977748 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:23.978231 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:23.979009 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:23.979225 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:23.979313 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:23.980793 (+ 1480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:49:23.980796 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:49:23.980798 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:49:23.980820 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:49:23.982261 (+ 1441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:49:23.982704 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:49:23.982707 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:49:23.982708 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:49:23.982781 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:49:23.983055 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:49:23.983057 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:49:23.983058 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:49:23.983242 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:49:23.983356 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:23.983583 (+ 227us) server_negotiation.cc:299] Negotiation successful
0812 09:49:23.983707 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260812 09:49:23.987111 12039 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50212:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 09:49:23.990147 12039 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 09:49:24.005014 12794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:23.998657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.3:46315 (local address 127.0.0.1:38938)
0812 09:49:23.999029 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:23.999079 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0812 09:49:23.999212 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:49:24.000246 (+ 1034us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:49:24.000252 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:49:24.000273 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:49:24.000419 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:49:24.000427 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:49:24.001950 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:49:24.001956 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:49:24.003289 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:49:24.003299 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:49:24.004416 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:49:24.004422 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:49:24.004558 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:24.004608 (+ 50us) client_negotiation.cc:769] Sending connection context
0812 09:49:24.004721 (+ 113us) client_negotiation.cc:240] Negotiation successful
0812 09:49:24.004848 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":77,"thread_start_us":81,"threads_started":1}
I20260812 09:49:24.005033 12796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:23.998839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38938 (local address 127.13.99.3:46315)
0812 09:49:23.999768 (+ 929us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:23.999780 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:23.999805 (+ 25us) server_negotiation.cc:407] Connection header received
0812 09:49:23.999887 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:23.999892 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:24.000107 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:24.000259 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:49:24.000597 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:24.001806 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:24.003418 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:24.004271 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:24.004397 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:24.004586 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:24.004767 (+ 181us) server_negotiation.cc:299] Negotiation successful
0812 09:49:24.004877 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":819,"thread_start_us":138,"threads_started":1}
I20260812 09:49:24.005076 12800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:23.999153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.2:33917 (local address 127.0.0.1:51986)
0812 09:49:23.999965 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:23.999978 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 09:49:24.000063 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:49:24.000713 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:49:24.000716 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:49:24.000735 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:49:24.000821 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:49:24.000828 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:49:24.002033 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:49:24.002037 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:49:24.003471 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:49:24.003483 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:49:24.004675 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:49:24.004678 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:49:24.004785 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:24.004808 (+ 23us) client_negotiation.cc:769] Sending connection context
0812 09:49:24.004881 (+ 73us) client_negotiation.cc:240] Negotiation successful
0812 09:49:24.004945 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"mutex_wait_us":58,"thread_start_us":86,"threads_started":1}
I20260812 09:49:24.005460 12802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:23.999577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51986 (local address 127.13.99.2:33917)
0812 09:49:24.000259 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:24.000271 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:24.000294 (+ 23us) server_negotiation.cc:407] Connection header received
0812 09:49:24.000370 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:24.000377 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:24.000582 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:24.000728 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:49:24.000922 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:24.001893 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:24.003603 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:24.004363 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:24.004452 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:24.004606 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:24.005228 (+ 622us) server_negotiation.cc:299] Negotiation successful
0812 09:49:24.005315 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":114,"threads_started":1}
I20260812 09:49:24.005584 12799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:23.999325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.1:38513 (local address 127.0.0.1:60538)
0812 09:49:24.000000 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:24.000014 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 09:49:24.000107 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:49:24.001001 (+ 894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:49:24.001004 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:49:24.001022 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:49:24.001121 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:49:24.001127 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:49:24.002521 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:49:24.002525 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:49:24.003653 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:49:24.003661 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:49:24.004540 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:49:24.004543 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:49:24.004624 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:24.004640 (+ 16us) client_negotiation.cc:769] Sending connection context
0812 09:49:24.004709 (+ 69us) client_negotiation.cc:240] Negotiation successful
0812 09:49:24.005471 (+ 762us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"mutex_wait_us":626,"thread_start_us":94,"threads_started":1}
I20260812 09:49:24.005874 12803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:23.999621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60538 (local address 127.13.99.1:38513)
0812 09:49:24.000365 (+ 744us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:24.000379 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:24.000421 (+ 42us) server_negotiation.cc:407] Connection header received
0812 09:49:24.000530 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:24.000534 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:24.000704 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:24.000831 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:49:24.001253 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:24.002363 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:24.003765 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:24.004326 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:24.004470 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:24.004922 (+ 452us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:24.004972 (+ 50us) server_negotiation.cc:299] Negotiation successful
0812 09:49:24.005071 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":646,"thread_start_us":78,"threads_started":1}
I20260812 09:49:24.008766 12235 tablet_service.cc:1467] Processing CreateTablet for tablet 6f79918e34ad43c281e4f5f3003cc4bf (DEFAULT_TABLE table=test-table [id=66587061685a46f2a4541fe977c68b6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 09:49:24.008626 12677 tablet_service.cc:1467] Processing CreateTablet for tablet 6f79918e34ad43c281e4f5f3003cc4bf (DEFAULT_TABLE table=test-table [id=66587061685a46f2a4541fe977c68b6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 09:49:24.008626 12448 tablet_service.cc:1467] Processing CreateTablet for tablet 6f79918e34ad43c281e4f5f3003cc4bf (DEFAULT_TABLE table=test-table [id=66587061685a46f2a4541fe977c68b6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 09:49:24.009703 12235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f79918e34ad43c281e4f5f3003cc4bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:49:24.009732 12448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f79918e34ad43c281e4f5f3003cc4bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:49:24.009732 12677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f79918e34ad43c281e4f5f3003cc4bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:49:24.017424 12806 tablet_bootstrap.cc:492] T 6f79918e34ad43c281e4f5f3003cc4bf P d264c1416cbd4e3aaf5af1d78159e779: Bootstrap starting.
I20260812 09:49:24.017421 12804 tablet_bootstrap.cc:492] T 6f79918e34ad43c281e4f5f3003cc4bf P 7a208411fc564d5591d4bf455ae2f0bf: Bootstrap starting.
I20260812 09:49:24.017830 12808 tablet_bootstrap.cc:492] T 6f79918e34ad43c281e4f5f3003cc4bf P 3c014875deec4513bbd5d4b1739cd4ae: Bootstrap starting.
I20260812 09:49:24.019822 12804 tablet_bootstrap.cc:654] T 6f79918e34ad43c281e4f5f3003cc4bf P 7a208411fc564d5591d4bf455ae2f0bf: Neither blocks nor log segments found. Creating new log.
I20260812 09:49:24.020275 12808 tablet_bootstrap.cc:654] T 6f79918e34ad43c281e4f5f3003cc4bf P 3c014875deec4513bbd5d4b1739cd4ae: Neither blocks nor log segments found. Creating new log.
I20260812 09:49:24.021238 12808 log.cc:824] T 6f79918e34ad43c281e4f5f3003cc4bf P 3c014875deec4513bbd5d4b1739cd4ae: Log is configured to *not* fsync() on all Append() calls
I20260812 09:49:24.021238 12804 log.cc:824] T 6f79918e34ad43c281e4f5f3003cc4bf P 7a208411fc564d5591d4bf455ae2f0bf: Log is configured to *not* fsync() on all Append() calls
I20260812 09:49:24.022104 12806 tablet_bootstrap.cc:654] T 6f79918e34ad43c281e4f5f3003cc4bf P d264c1416cbd4e3aaf5af1d78159e779: Neither blocks nor log segments found. Creating new log.
I20260812 09:49:24.023046 12806 log.cc:824] T 6f79918e34ad43c281e4f5f3003cc4bf P d264c1416cbd4e3aaf5af1d78159e779: Log is configured to *not* fsync() on all Append() calls
I20260812 09:49:24.023155 12808 tablet_bootstrap.cc:492] T 6f79918e34ad43c281e4f5f3003cc4bf P 3c014875deec4513bbd5d4b1739cd4ae: No bootstrap required, opened a new log
I20260812 09:49:24.023345 12808 ts_tablet_manager.cc:1397] T 6f79918e34ad43c281e4f5f3003cc4bf P 3c014875deec4513bbd5d4b1739cd4ae: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260812 09:49:24.024662 12804 tablet_bootstrap.cc:492] T 6f79918e34ad43c281e4f5f3003cc4bf P 7a208411fc564d5591d4bf455ae2f0bf: No bootstrap required, opened a new log
I20260812 09:49:24.024857 12804 ts_tablet_manager.cc:1397] T 6f79918e34ad43c281e4f5f3003cc4bf P 7a208411fc564d5591d4bf455ae2f0bf: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.004s
I20260812 09:49:24.025033 12806 tablet_bootstrap.cc:492] T 6f79918e34ad43c281e4f5f3003cc4bf P d264c1416cbd4e3aaf5af1d78159e779: No bootstrap required, opened a new log
I20260812 09:49:24.025210 12806 ts_tablet_manager.cc:1397] T 6f79918e34ad43c281e4f5f3003cc4bf P d264c1416cbd4e3aaf5af1d78159e779: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260812 09:49:24.026641 12808 raft_consensus.cc:348] T 6f79918e34ad43c281e4f5f3003cc4bf P 3c014875deec4513bbd5d4b1739cd4ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d264c1416cbd4e3aaf5af1d78159e779" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 46315 } } peers { permanent_uuid: "7a208411fc564d5591d4bf455ae2f0bf" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 38513 } } peers { permanent_uuid: "3c014875deec4513bbd5d4b1739cd4ae" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 33917 } }
I20260812 09:49:24.026885 12808 raft_consensus.cc:374] T 6f79918e34ad43c281e4f5f3003cc4bf P 3c014875deec4513bbd5d4b1739cd4ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:49:24.026970 12808 raft_consensus.cc:729] T 6f79918e34ad43c281e4f5f3003cc4bf P 3c014875deec4513bbd5d4b1739cd4ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c014875deec4513bbd5d4b1739cd4ae, State: Initialized, Role: FOLLOWER
I20260812 09:49:24.027374 12808 consensus_queue.cc:260] T 6f79918e34ad43c281e4f5f3003cc4bf P 3c014875deec4513bbd5d4b1739cd4ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d264c1416cbd4e3aaf5af1d78159e779" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 46315 } } peers { permanent_uuid: "7a208411fc564d5591d4bf455ae2f0bf" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 38513 } } peers { permanent_uuid: "3c014875deec4513bbd5d4b1739cd4ae" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 33917 } }
I20260812 09:49:24.028016 12554 heartbeater.cc:503] Master 127.13.99.62:41933 was elected leader, sending a full tablet report...
I20260812 09:49:24.028407 12808 ts_tablet_manager.cc:1428] T 6f79918e34ad43c281e4f5f3003cc4bf P 3c014875deec4513bbd5d4b1739cd4ae: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260812 09:49:24.028848 12804 raft_consensus.cc:348] T 6f79918e34ad43c281e4f5f3003cc4bf P 7a208411fc564d5591d4bf455ae2f0bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d264c1416cbd4e3aaf5af1d78159e779" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 46315 } } peers { permanent_uuid: "7a208411fc564d5591d4bf455ae2f0bf" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 38513 } } peers { permanent_uuid: "3c014875deec4513bbd5d4b1739cd4ae" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 33917 } }
I20260812 09:49:24.029132 12804 raft_consensus.cc:374] T 6f79918e34ad43c281e4f5f3003cc4bf P 7a208411fc564d5591d4bf455ae2f0bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:49:24.029197 12804 raft_consensus.cc:729] T 6f79918e34ad43c281e4f5f3003cc4bf P 7a208411fc564d5591d4bf455ae2f0bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a208411fc564d5591d4bf455ae2f0bf, State: Initialized, Role: FOLLOWER
I20260812 09:49:24.029633 12804 consensus_queue.cc:260] T 6f79918e34ad43c281e4f5f3003cc4bf P 7a208411fc564d5591d4bf455ae2f0bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d264c1416cbd4e3aaf5af1d78159e779" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 46315 } } peers { permanent_uuid: "7a208411fc564d5591d4bf455ae2f0bf" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 38513 } } peers { permanent_uuid: "3c014875deec4513bbd5d4b1739cd4ae" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 33917 } }
I20260812 09:49:24.029820 12806 raft_consensus.cc:348] T 6f79918e34ad43c281e4f5f3003cc4bf P d264c1416cbd4e3aaf5af1d78159e779 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d264c1416cbd4e3aaf5af1d78159e779" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 46315 } } peers { permanent_uuid: "7a208411fc564d5591d4bf455ae2f0bf" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 38513 } } peers { permanent_uuid: "3c014875deec4513bbd5d4b1739cd4ae" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 33917 } }
I20260812 09:49:24.030122 12806 raft_consensus.cc:374] T 6f79918e34ad43c281e4f5f3003cc4bf P d264c1416cbd4e3aaf5af1d78159e779 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:49:24.030208 12806 raft_consensus.cc:729] T 6f79918e34ad43c281e4f5f3003cc4bf P d264c1416cbd4e3aaf5af1d78159e779 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d264c1416cbd4e3aaf5af1d78159e779, State: Initialized, Role: FOLLOWER
I20260812 09:49:24.030488 12333 heartbeater.cc:503] Master 127.13.99.62:41933 was elected leader, sending a full tablet report...
I20260812 09:49:24.030611 12804 ts_tablet_manager.cc:1428] T 6f79918e34ad43c281e4f5f3003cc4bf P 7a208411fc564d5591d4bf455ae2f0bf: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260812 09:49:24.030651 12806 consensus_queue.cc:260] T 6f79918e34ad43c281e4f5f3003cc4bf P d264c1416cbd4e3aaf5af1d78159e779 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d264c1416cbd4e3aaf5af1d78159e779" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 46315 } } peers { permanent_uuid: "7a208411fc564d5591d4bf455ae2f0bf" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 38513 } } peers { permanent_uuid: "3c014875deec4513bbd5d4b1739cd4ae" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 33917 } }
I20260812 09:49:24.031294 12776 heartbeater.cc:503] Master 127.13.99.62:41933 was elected leader, sending a full tablet report...
I20260812 09:49:24.031563 12806 ts_tablet_manager.cc:1428] T 6f79918e34ad43c281e4f5f3003cc4bf P d264c1416cbd4e3aaf5af1d78159e779: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260812 09:49:24.162240 12822 raft_consensus.cc:482] T 6f79918e34ad43c281e4f5f3003cc4bf P 7a208411fc564d5591d4bf455ae2f0bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 09:49:24.162463 12822 raft_consensus.cc:504] T 6f79918e34ad43c281e4f5f3003cc4bf P 7a208411fc564d5591d4bf455ae2f0bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d264c1416cbd4e3aaf5af1d78159e779" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 46315 } } peers { permanent_uuid: "7a208411fc564d5591d4bf455ae2f0bf" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 38513 } } peers { permanent_uuid: "3c014875deec4513bbd5d4b1739cd4ae" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 33917 } }
I20260812 09:49:24.163695 12822 leader_election.cc:288] T 6f79918e34ad43c281e4f5f3003cc4bf P 7a208411fc564d5591d4bf455ae2f0bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d264c1416cbd4e3aaf5af1d78159e779 (127.13.99.3:46315), 3c014875deec4513bbd5d4b1739cd4ae (127.13.99.2:33917)
I20260812 09:49:24.169549 12802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:24.164114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.1:46317 (local address 127.13.99.2:33917)
0812 09:49:24.164247 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:24.164252 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:24.164510 (+ 258us) server_negotiation.cc:407] Connection header received
0812 09:49:24.164572 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:24.164575 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:24.164632 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:24.164725 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:49:24.165297 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:24.166090 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:24.168092 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:24.168780 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:24.168914 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:24.168983 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:24.169362 (+ 379us) server_negotiation.cc:299] Negotiation successful
0812 09:49:24.169432 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260812 09:49:24.169557 12796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:24.164046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.1:43465 (local address 127.13.99.3:46315)
0812 09:49:24.164181 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:24.164185 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:24.164422 (+ 237us) server_negotiation.cc:407] Connection header received
0812 09:49:24.164573 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:24.164575 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:24.164632 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:24.164723 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:49:24.165296 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:24.166107 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:24.168090 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:24.168780 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:24.168938 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:24.169013 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:24.169340 (+ 327us) server_negotiation.cc:299] Negotiation successful
0812 09:49:24.169414 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260812 09:49:24.169677 12830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:24.164003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.2:33917 (local address 127.13.99.1:46317)
0812 09:49:24.164362 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:24.164381 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 09:49:24.164466 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:49:24.164757 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:49:24.164760 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:49:24.164805 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:49:24.165164 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:49:24.165172 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:49:24.166246 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:49:24.166250 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:49:24.167921 (+ 1671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:49:24.167934 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:49:24.168951 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:49:24.168955 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:49:24.169168 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:24.169220 (+ 52us) client_negotiation.cc:769] Sending connection context
0812 09:49:24.169307 (+ 87us) client_negotiation.cc:240] Negotiation successful
0812 09:49:24.169475 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"mutex_wait_us":377,"spinlock_wait_cycles":7424,"thread_start_us":112,"threads_started":1}
I20260812 09:49:24.169674 12829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:24.163894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.3:46315 (local address 127.13.99.1:43465)
0812 09:49:24.164286 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:24.164301 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 09:49:24.164445 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:49:24.164757 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:49:24.164760 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:49:24.164804 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:49:24.165165 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:49:24.165172 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:49:24.166246 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:49:24.166250 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:49:24.167922 (+ 1672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:49:24.167934 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:49:24.168951 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:49:24.168955 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:49:24.169172 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:24.169224 (+ 52us) client_negotiation.cc:769] Sending connection context
0812 09:49:24.169307 (+ 83us) client_negotiation.cc:240] Negotiation successful
0812 09:49:24.169450 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"mutex_wait_us":391,"spinlock_wait_cycles":33024,"thread_start_us":124,"threads_started":1}
I20260812 09:49:24.170403 12704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f79918e34ad43c281e4f5f3003cc4bf" candidate_uuid: "7a208411fc564d5591d4bf455ae2f0bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d264c1416cbd4e3aaf5af1d78159e779" is_pre_election: true
I20260812 09:49:24.170413 12481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f79918e34ad43c281e4f5f3003cc4bf" candidate_uuid: "7a208411fc564d5591d4bf455ae2f0bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c014875deec4513bbd5d4b1739cd4ae" is_pre_election: true
I20260812 09:49:24.170706 12704 raft_consensus.cc:2454] T 6f79918e34ad43c281e4f5f3003cc4bf P d264c1416cbd4e3aaf5af1d78159e779 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a208411fc564d5591d4bf455ae2f0bf in term 0.
I20260812 09:49:24.170732 12481 raft_consensus.cc:2454] T 6f79918e34ad43c281e4f5f3003cc4bf P 3c014875deec4513bbd5d4b1739cd4ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a208411fc564d5591d4bf455ae2f0bf in term 0.
I20260812 09:49:24.171259 12169 leader_election.cc:302] T 6f79918e34ad43c281e4f5f3003cc4bf P 7a208411fc564d5591d4bf455ae2f0bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3c014875deec4513bbd5d4b1739cd4ae, 7a208411fc564d5591d4bf455ae2f0bf, d264c1416cbd4e3aaf5af1d78159e779; no voters:
I20260812 09:49:24.171561 12822 raft_consensus.cc:2781] T 6f79918e34ad43c281e4f5f3003cc4bf P 7a208411fc564d5591d4bf455ae2f0bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 09:49:24.171643 12822 raft_consensus.cc:482] T 6f79918e34ad43c281e4f5f3003cc4bf P 7a208411fc564d5591d4bf455ae2f0bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 09:49:24.171675 12822 raft_consensus.cc:3037] T 6f79918e34ad43c281e4f5f3003cc4bf P 7a208411fc564d5591d4bf455ae2f0bf [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:49:24.172851 12822 raft_consensus.cc:504] T 6f79918e34ad43c281e4f5f3003cc4bf P 7a208411fc564d5591d4bf455ae2f0bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d264c1416cbd4e3aaf5af1d78159e779" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 46315 } } peers { permanent_uuid: "7a208411fc564d5591d4bf455ae2f0bf" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 38513 } } peers { permanent_uuid: "3c014875deec4513bbd5d4b1739cd4ae" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 33917 } }
I20260812 09:49:24.173246 12822 leader_election.cc:288] T 6f79918e34ad43c281e4f5f3003cc4bf P 7a208411fc564d5591d4bf455ae2f0bf [CANDIDATE]: Term 1 election: Requested vote from peers d264c1416cbd4e3aaf5af1d78159e779 (127.13.99.3:46315), 3c014875deec4513bbd5d4b1739cd4ae (127.13.99.2:33917)
I20260812 09:49:24.173573 12704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f79918e34ad43c281e4f5f3003cc4bf" candidate_uuid: "7a208411fc564d5591d4bf455ae2f0bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d264c1416cbd4e3aaf5af1d78159e779"
I20260812 09:49:24.173676 12704 raft_consensus.cc:3037] T 6f79918e34ad43c281e4f5f3003cc4bf P d264c1416cbd4e3aaf5af1d78159e779 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:49:24.173691 12481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f79918e34ad43c281e4f5f3003cc4bf" candidate_uuid: "7a208411fc564d5591d4bf455ae2f0bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c014875deec4513bbd5d4b1739cd4ae"
I20260812 09:49:24.173834 12481 raft_consensus.cc:3037] T 6f79918e34ad43c281e4f5f3003cc4bf P 3c014875deec4513bbd5d4b1739cd4ae [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:49:24.174813 12704 raft_consensus.cc:2454] T 6f79918e34ad43c281e4f5f3003cc4bf P d264c1416cbd4e3aaf5af1d78159e779 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a208411fc564d5591d4bf455ae2f0bf in term 1.
I20260812 09:49:24.174851 12481 raft_consensus.cc:2454] T 6f79918e34ad43c281e4f5f3003cc4bf P 3c014875deec4513bbd5d4b1739cd4ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a208411fc564d5591d4bf455ae2f0bf in term 1.
I20260812 09:49:24.175150 12167 leader_election.cc:302] T 6f79918e34ad43c281e4f5f3003cc4bf P 7a208411fc564d5591d4bf455ae2f0bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a208411fc564d5591d4bf455ae2f0bf, d264c1416cbd4e3aaf5af1d78159e779; no voters:
I20260812 09:49:24.175400 12822 raft_consensus.cc:2781] T 6f79918e34ad43c281e4f5f3003cc4bf P 7a208411fc564d5591d4bf455ae2f0bf [term 1 FOLLOWER]: Leader election won for term 1
I20260812 09:49:24.175588 12822 raft_consensus.cc:686] T 6f79918e34ad43c281e4f5f3003cc4bf P 7a208411fc564d5591d4bf455ae2f0bf [term 1 LEADER]: Becoming Leader. State: Replica: 7a208411fc564d5591d4bf455ae2f0bf, State: Running, Role: LEADER
I20260812 09:49:24.175915 12822 consensus_queue.cc:237] T 6f79918e34ad43c281e4f5f3003cc4bf P 7a208411fc564d5591d4bf455ae2f0bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d264c1416cbd4e3aaf5af1d78159e779" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 46315 } } peers { permanent_uuid: "7a208411fc564d5591d4bf455ae2f0bf" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 38513 } } peers { permanent_uuid: "3c014875deec4513bbd5d4b1739cd4ae" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 33917 } }
I20260812 09:49:24.179127 12039 catalog_manager.cc:5495] T 6f79918e34ad43c281e4f5f3003cc4bf P 7a208411fc564d5591d4bf455ae2f0bf reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a208411fc564d5591d4bf455ae2f0bf (127.13.99.1). New cstate: current_term: 1 leader_uuid: "7a208411fc564d5591d4bf455ae2f0bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d264c1416cbd4e3aaf5af1d78159e779" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 46315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a208411fc564d5591d4bf455ae2f0bf" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 38513 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c014875deec4513bbd5d4b1739cd4ae" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 33917 } health_report { overall_health: UNKNOWN } } }
I20260812 09:49:24.193576 12803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:24.190218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60540 (local address 127.13.99.1:38513)
0812 09:49:24.190342 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:24.190345 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:24.190360 (+ 15us) server_negotiation.cc:407] Connection header received
0812 09:49:24.190443 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:24.190445 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:24.190496 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:24.190581 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 09:49:24.190845 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:24.191378 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:24.192185 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:24.192397 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:24.192541 (+ 144us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:24.192811 (+ 270us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 09:49:24.193191 (+ 380us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 09:49:24.193300 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:24.193435 (+ 135us) server_negotiation.cc:299] Negotiation successful
0812 09:49:24.193470 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260812 09:49:24.199570 12704 raft_consensus.cc:1261] T 6f79918e34ad43c281e4f5f3003cc4bf P d264c1416cbd4e3aaf5af1d78159e779 [term 1 FOLLOWER]: Refusing update from remote peer 7a208411fc564d5591d4bf455ae2f0bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 09:49:24.199570 12481 raft_consensus.cc:1261] T 6f79918e34ad43c281e4f5f3003cc4bf P 3c014875deec4513bbd5d4b1739cd4ae [term 1 FOLLOWER]: Refusing update from remote peer 7a208411fc564d5591d4bf455ae2f0bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 09:49:24.200366 12831 consensus_queue.cc:1035] T 6f79918e34ad43c281e4f5f3003cc4bf P 7a208411fc564d5591d4bf455ae2f0bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "d264c1416cbd4e3aaf5af1d78159e779" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 46315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 09:49:24.200578 12822 consensus_queue.cc:1035] T 6f79918e34ad43c281e4f5f3003cc4bf P 7a208411fc564d5591d4bf455ae2f0bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c014875deec4513bbd5d4b1739cd4ae" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 33917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 09:49:24.204972 12779 tablet.cc:2369] T 6f79918e34ad43c281e4f5f3003cc4bf P d264c1416cbd4e3aaf5af1d78159e779: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 09:49:24.207962 12838 mvcc.cc:205] Tried to move back new op lower bound from 7317619360553500672 to 7317619360467476480. Current Snapshot: MvccSnapshot[applied={T|T < 7317619360553500672}]
I20260812 09:49:24.209813 12840 mvcc.cc:205] Tried to move back new op lower bound from 7317619360553500672 to 7317619360467476480. Current Snapshot: MvccSnapshot[applied={T|T < 7317619360553500672}]
I20260812 09:49:24.216826 12039 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50212:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 09:49:24.217015 12039 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50212:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 09:49:24.219451 12039 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a0a5607944dc48b9acf07d32f2f4f32b: Sending DeleteTablet for 3 replicas of tablet 6f79918e34ad43c281e4f5f3003cc4bf
I20260812 09:49:24.220213 12677 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f79918e34ad43c281e4f5f3003cc4bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 09:49:24 UTC) from {username='kudu', principal='kudu/127.13.99.62@KRBTEST.COM'} at 127.0.0.1:38938
I20260812 09:49:24.220294 12448 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f79918e34ad43c281e4f5f3003cc4bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 09:49:24 UTC) from {username='kudu', principal='kudu/127.13.99.62@KRBTEST.COM'} at 127.0.0.1:51986
I20260812 09:49:24.220657 12235 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f79918e34ad43c281e4f5f3003cc4bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 09:49:24 UTC) from {username='kudu', principal='kudu/127.13.99.62@KRBTEST.COM'} at 127.0.0.1:60538
I20260812 09:49:24.221127 12850 tablet_replica.cc:330] T 6f79918e34ad43c281e4f5f3003cc4bf P 7a208411fc564d5591d4bf455ae2f0bf: stopping tablet replica
I20260812 09:49:24.221217 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 12140
I20260812 09:49:24.221501 12850 raft_consensus.cc:2229] T 6f79918e34ad43c281e4f5f3003cc4bf P 7a208411fc564d5591d4bf455ae2f0bf [term 1 LEADER]: Raft consensus shutting down.
I20260812 09:49:24.224397 12849 tablet_replica.cc:330] T 6f79918e34ad43c281e4f5f3003cc4bf P 3c014875deec4513bbd5d4b1739cd4ae: stopping tablet replica
I20260812 09:49:24.224756 12849 raft_consensus.cc:2229] T 6f79918e34ad43c281e4f5f3003cc4bf P 3c014875deec4513bbd5d4b1739cd4ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 09:49:24.225077 12849 raft_consensus.cc:2258] T 6f79918e34ad43c281e4f5f3003cc4bf P 3c014875deec4513bbd5d4b1739cd4ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 09:49:24.227226 12851 tablet_replica.cc:330] T 6f79918e34ad43c281e4f5f3003cc4bf P d264c1416cbd4e3aaf5af1d78159e779: stopping tablet replica
I20260812 09:49:24.227676 12851 raft_consensus.cc:2229] T 6f79918e34ad43c281e4f5f3003cc4bf P d264c1416cbd4e3aaf5af1d78159e779 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 09:49:24.228127 12851 raft_consensus.cc:2258] T 6f79918e34ad43c281e4f5f3003cc4bf P d264c1416cbd4e3aaf5af1d78159e779 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 09:49:24.229492 12849 ts_tablet_manager.cc:1905] T 6f79918e34ad43c281e4f5f3003cc4bf P 3c014875deec4513bbd5d4b1739cd4ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 09:49:24.232786 12851 ts_tablet_manager.cc:1905] T 6f79918e34ad43c281e4f5f3003cc4bf P d264c1416cbd4e3aaf5af1d78159e779: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 09:49:24.233421 12849 ts_tablet_manager.cc:1918] T 6f79918e34ad43c281e4f5f3003cc4bf P 3c014875deec4513bbd5d4b1739cd4ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 09:49:24.233536 12849 log.cc:1196] T 6f79918e34ad43c281e4f5f3003cc4bf P 3c014875deec4513bbd5d4b1739cd4ae: Deleting WAL directory at /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/minicluster-data/ts-1/wal/wals/6f79918e34ad43c281e4f5f3003cc4bf
I20260812 09:49:24.233963 12849 ts_tablet_manager.cc:1939] T 6f79918e34ad43c281e4f5f3003cc4bf P 3c014875deec4513bbd5d4b1739cd4ae: Deleting consensus metadata
I20260812 09:49:24.235035 12851 ts_tablet_manager.cc:1918] T 6f79918e34ad43c281e4f5f3003cc4bf P d264c1416cbd4e3aaf5af1d78159e779: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 09:49:24.235294 12851 log.cc:1196] T 6f79918e34ad43c281e4f5f3003cc4bf P d264c1416cbd4e3aaf5af1d78159e779: Deleting WAL directory at /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786527905902985-13708-0/minicluster-data/ts-2/wal/wals/6f79918e34ad43c281e4f5f3003cc4bf
I20260812 09:49:24.235368 12019 catalog_manager.cc:4841] TS 3c014875deec4513bbd5d4b1739cd4ae (127.13.99.2:33917): tablet 6f79918e34ad43c281e4f5f3003cc4bf (table test-table [id=66587061685a46f2a4541fe977c68b6d]) successfully deleted
I20260812 09:49:24.235677 12851 ts_tablet_manager.cc:1939] T 6f79918e34ad43c281e4f5f3003cc4bf P d264c1416cbd4e3aaf5af1d78159e779: Deleting consensus metadata
I20260812 09:49:24.236701 12017 catalog_manager.cc:4841] TS d264c1416cbd4e3aaf5af1d78159e779 (127.13.99.3:46315): tablet 6f79918e34ad43c281e4f5f3003cc4bf (table test-table [id=66587061685a46f2a4541fe977c68b6d]) successfully deleted
W20260812 09:49:24.240082 12017 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.99.1:38513 (error 108)
I20260812 09:49:24.240276 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 12343
I20260812 09:49:24.251139 12799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:24.244512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.1:38513 (local address 127.0.0.1:60550)
0812 09:49:24.250970 (+ 6458us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:24.251055 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.1:38513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":6237}
W20260812 09:49:24.253348 12017 catalog_manager.cc:4584] TS 7a208411fc564d5591d4bf455ae2f0bf (127.13.99.1:38513): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6f79918e34ad43c281e4f5f3003cc4bf: Network error: Client connection negotiation failed: client connection to 127.13.99.1:38513: connect: Connection refused (error 111)
I20260812 09:49:24.260104 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 12562
I20260812 09:49:24.290426 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 11992
2026-08-12T09:49:24Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2662 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-12T09:49:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T09:49:24Z Disabled control of system clock
I20260812 09:49:24.331583 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786527905902985-13708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786527905902985-13708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786527905902985-13708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.62:41837
--webserver_interface=127.13.99.62
--webserver_port=0
--builtin_ntp_servers=127.13.99.20:43057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.99.62:41837
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 09:49:24.445873 12858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:49:24.446121 12858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:49:24.446156 12858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:49:24.450121 12858 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 09:49:24.450177 12858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:49:24.450193 12858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:49:24.450210 12858 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 09:49:24.450225 12858 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 09:49:24.454872 12858 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:43057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786527905902985-13708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786527905902985-13708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.99.62:41837
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.62:41837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.99.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12858
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786527905902985-13708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:49:24.455850 12858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:49:24.457317 12858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:49:24.465777 12871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:49:24.465838 12868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:49:24.465816 12869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:49:24.465981 12858 server_base.cc:1034] running on GCE node
I20260812 09:49:24.466756 12858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:49:24.467832 12858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:49:24.469027 12858 hybrid_clock.cc:648] HybridClock initialized: now 1786528164469016 us; error 28 us; skew 500 ppm
I20260812 09:49:24.470841 12858 webserver.cc:460] Webserver started at http://127.13.99.62:45869/ using document root <none> and password file <none>
I20260812 09:49:24.471472 12858 fs_manager.cc:359] Metadata directory not provided
I20260812 09:49:24.471572 12858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:49:24.471796 12858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:49:24.473907 12858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786527905902985-13708-0/minicluster-data/master-0/data/instance:
uuid: "3fc106fe4c55423597446f25e9db4863"
format_stamp: "Formatted at 2026-08-12 09:49:24 on dist-test-slave-jq4t"
server_key: "56a3c30ee761b02dc4ba2fd141f0bd97"
server_key_iv: "ff5e131319deff7e3510a9f4879a0d06"
server_key_version: "encryptionkey@0"
I20260812 09:49:24.474501 12858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786527905902985-13708-0/minicluster-data/master-0/wal/instance:
uuid: "3fc106fe4c55423597446f25e9db4863"
format_stamp: "Formatted at 2026-08-12 09:49:24 on dist-test-slave-jq4t"
server_key: "56a3c30ee761b02dc4ba2fd141f0bd97"
server_key_iv: "ff5e131319deff7e3510a9f4879a0d06"
server_key_version: "encryptionkey@0"
I20260812 09:49:24.478245 12858 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260812 09:49:24.480721 12879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:49:24.481917 12858 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260812 09:49:24.482075 12858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786527905902985-13708-0/minicluster-data/master-0/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786527905902985-13708-0/minicluster-data/master-0/wal
uuid: "3fc106fe4c55423597446f25e9db4863"
format_stamp: "Formatted at 2026-08-12 09:49:24 on dist-test-slave-jq4t"
server_key: "56a3c30ee761b02dc4ba2fd141f0bd97"
server_key_iv: "ff5e131319deff7e3510a9f4879a0d06"
server_key_version: "encryptionkey@0"
I20260812 09:49:24.482189 12858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786527905902985-13708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786527905902985-13708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786527905902985-13708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:49:24.498468 12858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:49:24.499155 12858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:49:24.499357 12858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:49:24.506760 12858 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.62:41837
I20260812 09:49:24.506768 12967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.62:41837 every 8 connection(s)
I20260812 09:49:24.507752 12858 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
I20260812 09:49:24.511006 12968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:49:24.516755 12968 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fc106fe4c55423597446f25e9db4863: Bootstrap starting.
I20260812 09:49:24.517824 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 12858
I20260812 09:49:24.517972 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786527905902985-13708-0/minicluster-data/master-0/wal/instance
I20260812 09:49:24.518283 13708 external_mini_cluster.cc:1428] Setting key 7c89e924cd4b9a07ee9005fb6bda97bd
I20260812 09:49:24.519676 12968 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3fc106fe4c55423597446f25e9db4863: Neither blocks nor log segments found. Creating new log.
I20260812 09:49:24.520468 12968 log.cc:824] T 00000000000000000000000000000000 P 3fc106fe4c55423597446f25e9db4863: Log is configured to *not* fsync() on all Append() calls
I20260812 09:49:24.522733 12968 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fc106fe4c55423597446f25e9db4863: No bootstrap required, opened a new log
I20260812 09:49:24.527153 12968 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3fc106fe4c55423597446f25e9db4863 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fc106fe4c55423597446f25e9db4863" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 41837 } }
I20260812 09:49:24.527398 12968 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3fc106fe4c55423597446f25e9db4863 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:49:24.527493 12968 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3fc106fe4c55423597446f25e9db4863 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fc106fe4c55423597446f25e9db4863, State: Initialized, Role: FOLLOWER
I20260812 09:49:24.527973 12979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:24.519550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54958 (local address 127.13.99.62:41837)
0812 09:49:24.520280 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:24.520295 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:24.520330 (+ 35us) server_negotiation.cc:407] Connection header received
0812 09:49:24.521111 (+ 781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:24.521130 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:24.521461 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:24.521742 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:49:24.522099 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:24.523155 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:24.524361 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:24.524968 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:24.525111 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:24.526070 (+ 959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:49:24.526075 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:49:24.526088 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 09:49:24.526477 (+ 389us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:49:24.526534 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 09:49:24.526554 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:49:24.526636 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:24.526809 (+ 173us) server_negotiation.cc:299] Negotiation successful
0812 09:49:24.527124 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":190,"threads_started":1}
I20260812 09:49:24.527978 12968 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3fc106fe4c55423597446f25e9db4863 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fc106fe4c55423597446f25e9db4863" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 41837 } }
I20260812 09:49:24.528235 12968 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3fc106fe4c55423597446f25e9db4863 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 09:49:24.528296 12968 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3fc106fe4c55423597446f25e9db4863 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 09:49:24.528388 12968 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3fc106fe4c55423597446f25e9db4863 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:49:24.529453 12968 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3fc106fe4c55423597446f25e9db4863 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fc106fe4c55423597446f25e9db4863" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 41837 } }
I20260812 09:49:24.529795 12968 leader_election.cc:302] T 00000000000000000000000000000000 P 3fc106fe4c55423597446f25e9db4863 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3fc106fe4c55423597446f25e9db4863; no voters:
I20260812 09:49:24.530138 12968 leader_election.cc:288] T 00000000000000000000000000000000 P 3fc106fe4c55423597446f25e9db4863 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 09:49:24.530258 12983 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3fc106fe4c55423597446f25e9db4863 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 09:49:24.530493 12983 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3fc106fe4c55423597446f25e9db4863 [term 1 LEADER]: Becoming Leader. State: Replica: 3fc106fe4c55423597446f25e9db4863, State: Running, Role: LEADER
I20260812 09:49:24.530833 12983 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3fc106fe4c55423597446f25e9db4863 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fc106fe4c55423597446f25e9db4863" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 41837 } }
I20260812 09:49:24.531420 12968 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3fc106fe4c55423597446f25e9db4863 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 09:49:24.532428 12984 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fc106fe4c55423597446f25e9db4863 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3fc106fe4c55423597446f25e9db4863" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fc106fe4c55423597446f25e9db4863" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 41837 } } }
I20260812 09:49:24.532580 12984 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fc106fe4c55423597446f25e9db4863 [sys.catalog]: This master's current role is: LEADER
I20260812 09:49:24.532940 12985 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fc106fe4c55423597446f25e9db4863 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3fc106fe4c55423597446f25e9db4863. Latest consensus state: current_term: 1 leader_uuid: "3fc106fe4c55423597446f25e9db4863" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fc106fe4c55423597446f25e9db4863" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 41837 } } }
I20260812 09:49:24.533046 12985 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fc106fe4c55423597446f25e9db4863 [sys.catalog]: This master's current role is: LEADER
I20260812 09:49:24.533387 12999 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 09:49:24.536051 12999 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 09:49:24.541237 12999 catalog_manager.cc:1347] Generated new cluster ID: b116c6a62a7c476a87d76833632c78f4
I20260812 09:49:24.541302 12999 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 09:49:24.549275 12999 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 09:49:24.550166 12999 catalog_manager.cc:1504] Loading token signing keys...
I20260812 09:49:24.555176 12999 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3fc106fe4c55423597446f25e9db4863: Generated new TSK 0
I20260812 09:49:24.555800 12999 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 09:49:24.564098 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786527905902985-13708-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786527905902985-13708-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786527905902985-13708-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786527905902985-13708-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.99.1:0
--local_ip_for_outbound_sockets=127.13.99.1
--webserver_interface=127.13.99.1
--webserver_port=0
--tserver_master_addrs=127.13.99.62:41837
--builtin_ntp_servers=127.13.99.20:43057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 09:49:24.691946 13015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:49:24.692212 13015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:49:24.692255 13015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:49:24.696265 13015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:49:24.696323 13015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:49:24.696378 13015 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.99.1
I20260812 09:49:24.701116 13015 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:43057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786527905902985-13708-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786527905902985-13708-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.99.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786527905902985-13708-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.99.1
--webserver_port=0
--tserver_master_addrs=127.13.99.62:41837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13015
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.99.1
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786527905902985-13708-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:49:24.702256 13015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:49:24.703797 13015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:49:24.713932 13021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:49:24.713939 13022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:49:24.713990 13015 server_base.cc:1034] running on GCE node
W20260812 09:49:24.713939 13026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:49:24.714515 13015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:49:24.715159 13015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:49:24.716351 13015 hybrid_clock.cc:648] HybridClock initialized: now 1786528164716334 us; error 29 us; skew 500 ppm
I20260812 09:49:24.718202 13015 webserver.cc:460] Webserver started at http://127.13.99.1:46587/ using document root <none> and password file <none>
I20260812 09:49:24.718786 13015 fs_manager.cc:359] Metadata directory not provided
I20260812 09:49:24.718878 13015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:49:24.719113 13015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:49:24.721246 13015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786527905902985-13708-0/minicluster-data/ts-0/data/instance:
uuid: "fb2fb1cac5a94c5ebd6eb37dd31ef391"
format_stamp: "Formatted at 2026-08-12 09:49:24 on dist-test-slave-jq4t"
server_key: "c7782b5154e86fbed444fe57147811ad"
server_key_iv: "cd2bed82ab8495d23f6a145f9ce1d0e2"
server_key_version: "encryptionkey@0"
I20260812 09:49:24.721814 13015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786527905902985-13708-0/minicluster-data/ts-0/wal/instance:
uuid: "fb2fb1cac5a94c5ebd6eb37dd31ef391"
format_stamp: "Formatted at 2026-08-12 09:49:24 on dist-test-slave-jq4t"
server_key: "c7782b5154e86fbed444fe57147811ad"
server_key_iv: "cd2bed82ab8495d23f6a145f9ce1d0e2"
server_key_version: "encryptionkey@0"
I20260812 09:49:24.725452 13015 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 09:49:24.727873 13033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:49:24.729126 13015 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260812 09:49:24.729255 13015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786527905902985-13708-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786527905902985-13708-0/minicluster-data/ts-0/wal
uuid: "fb2fb1cac5a94c5ebd6eb37dd31ef391"
format_stamp: "Formatted at 2026-08-12 09:49:24 on dist-test-slave-jq4t"
server_key: "c7782b5154e86fbed444fe57147811ad"
server_key_iv: "cd2bed82ab8495d23f6a145f9ce1d0e2"
server_key_version: "encryptionkey@0"
I20260812 09:49:24.729362 13015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786527905902985-13708-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786527905902985-13708-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786527905902985-13708-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:49:24.741874 13015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:49:24.742548 13015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:49:24.742739 13015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:49:24.743319 13015 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 09:49:24.744261 13015 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:49:24.744333 13015 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 09:49:24.744395 13015 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:49:24.744436 13015 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 09:49:24.753443 13015 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.1:46507
I20260812 09:49:24.753520 13217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.1:46507 every 8 connection(s)
I20260812 09:49:24.754402 13015 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786527905902985-13708-0/minicluster-data/ts-0/data/info.pb
I20260812 09:49:24.763055 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 13015
I20260812 09:49:24.763214 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786527905902985-13708-0/minicluster-data/ts-0/wal/instance
I20260812 09:49:24.763499 12979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:24.756280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.1:56613 (local address 127.13.99.62:41837)
0812 09:49:24.756428 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:24.756432 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:24.757571 (+ 1139us) server_negotiation.cc:407] Connection header received
0812 09:49:24.758927 (+ 1356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:24.758930 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:24.758975 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:24.759056 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:49:24.759936 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:24.760503 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:24.761606 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:24.761820 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:24.761897 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:24.762986 (+ 1089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:49:24.762989 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:49:24.762991 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 09:49:24.763003 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:49:24.763020 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 09:49:24.763028 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:49:24.763108 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:24.763292 (+ 184us) server_negotiation.cc:299] Negotiation successful
0812 09:49:24.763354 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260812 09:49:24.763518 13708 external_mini_cluster.cc:1428] Setting key ed52017b7ec24594fe6ed47d3e523b87
I20260812 09:49:24.764562 13220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:24.756766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:41837 (local address 127.13.99.1:56613)
0812 09:49:24.757429 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:24.757469 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0812 09:49:24.758737 (+ 1268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:49:24.759225 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:49:24.759231 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:49:24.759310 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:49:24.759813 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:49:24.759826 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:49:24.760609 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:49:24.760612 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:49:24.761500 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:49:24.761509 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:49:24.761951 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:49:24.761954 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:49:24.762097 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:24.762692 (+ 595us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 09:49:24.762693 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:49:24.762732 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 09:49:24.762810 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:49:24.763123 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:49:24.763135 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:49:24.763141 (+ 6us) client_negotiation.cc:769] Sending connection context
0812 09:49:24.763288 (+ 147us) client_negotiation.cc:240] Negotiation successful
0812 09:49:24.763578 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":114,"threads_started":1}
I20260812 09:49:24.765658 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786527905902985-13708-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786527905902985-13708-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786527905902985-13708-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786527905902985-13708-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.99.2:0
--local_ip_for_outbound_sockets=127.13.99.2
--webserver_interface=127.13.99.2
--webserver_port=0
--tserver_master_addrs=127.13.99.62:41837
--builtin_ntp_servers=127.13.99.20:43057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 09:49:24.765861 13218 heartbeater.cc:347] Connected to a master server at 127.13.99.62:41837
I20260812 09:49:24.766114 13218 heartbeater.cc:464] Registering TS with master...
I20260812 09:49:24.766654 13218 heartbeater.cc:511] Master 127.13.99.62:41837 requested a full tablet report, sending...
I20260812 09:49:24.768217 12910 ts_manager.cc:194] Registered new tserver with Master: fb2fb1cac5a94c5ebd6eb37dd31ef391 (127.13.99.1:46507)
I20260812 09:49:24.769428 12910 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.99.1:56613
W20260812 09:49:24.879472 13221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:49:24.879758 13221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:49:24.879822 13221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:49:24.883781 13221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:49:24.883864 13221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:49:24.883957 13221 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.99.2
I20260812 09:49:24.888830 13221 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:43057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786527905902985-13708-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786527905902985-13708-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.99.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786527905902985-13708-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.99.2
--webserver_port=0
--tserver_master_addrs=127.13.99.62:41837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13221
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.99.2
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786527905902985-13708-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:49:24.889953 13221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:49:24.891410 13221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:49:24.901237 13236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:49:24.901232 13237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:49:24.901237 13242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:49:24.901623 13221 server_base.cc:1034] running on GCE node
I20260812 09:49:24.902047 13221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:49:24.902693 13221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:49:24.903896 13221 hybrid_clock.cc:648] HybridClock initialized: now 1786528164903883 us; error 67 us; skew 500 ppm
I20260812 09:49:24.905941 13221 webserver.cc:460] Webserver started at http://127.13.99.2:45469/ using document root <none> and password file <none>
I20260812 09:49:24.906544 13221 fs_manager.cc:359] Metadata directory not provided
I20260812 09:49:24.906639 13221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:49:24.906876 13221 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:49:24.909039 13221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786527905902985-13708-0/minicluster-data/ts-1/data/instance:
uuid: "da4f7e8e0df5467995d220b56b23ca48"
format_stamp: "Formatted at 2026-08-12 09:49:24 on dist-test-slave-jq4t"
server_key: "ac18b741adca8fba8069fe04c21bed52"
server_key_iv: "582b218d272729640654959b6abf799c"
server_key_version: "encryptionkey@0"
I20260812 09:49:24.909619 13221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786527905902985-13708-0/minicluster-data/ts-1/wal/instance:
uuid: "da4f7e8e0df5467995d220b56b23ca48"
format_stamp: "Formatted at 2026-08-12 09:49:24 on dist-test-slave-jq4t"
server_key: "ac18b741adca8fba8069fe04c21bed52"
server_key_iv: "582b218d272729640654959b6abf799c"
server_key_version: "encryptionkey@0"
I20260812 09:49:24.913313 13221 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260812 09:49:24.915705 13251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:49:24.917013 13221 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260812 09:49:24.917157 13221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786527905902985-13708-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786527905902985-13708-0/minicluster-data/ts-1/wal
uuid: "da4f7e8e0df5467995d220b56b23ca48"
format_stamp: "Formatted at 2026-08-12 09:49:24 on dist-test-slave-jq4t"
server_key: "ac18b741adca8fba8069fe04c21bed52"
server_key_iv: "582b218d272729640654959b6abf799c"
server_key_version: "encryptionkey@0"
I20260812 09:49:24.917272 13221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786527905902985-13708-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786527905902985-13708-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786527905902985-13708-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:49:24.931526 13221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:49:24.932446 13221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:49:24.932747 13221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:49:24.933466 13221 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 09:49:24.934517 13221 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:49:24.934585 13221 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 09:49:24.934649 13221 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:49:24.934679 13221 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 09:49:24.944075 13221 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.2:39901
I20260812 09:49:24.944105 13457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.2:39901 every 8 connection(s)
I20260812 09:49:24.945061 13221 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786527905902985-13708-0/minicluster-data/ts-1/data/info.pb
I20260812 09:49:24.953318 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 13221
I20260812 09:49:24.953464 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786527905902985-13708-0/minicluster-data/ts-1/wal/instance
I20260812 09:49:24.953830 13708 external_mini_cluster.cc:1428] Setting key 86329d6b87e0a590aa43d42ee831c778
I20260812 09:49:24.954674 12979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:24.946943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.2:36177 (local address 127.13.99.62:41837)
0812 09:49:24.947123 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:24.947129 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:24.948208 (+ 1079us) server_negotiation.cc:407] Connection header received
0812 09:49:24.949660 (+ 1452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:24.949663 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:24.949713 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:24.949792 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:49:24.950746 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:24.951302 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:24.952533 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:24.952751 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:24.952831 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:24.954161 (+ 1330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:49:24.954164 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:49:24.954166 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 09:49:24.954179 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:49:24.954197 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 09:49:24.954205 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:49:24.954264 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:24.954466 (+ 202us) server_negotiation.cc:299] Negotiation successful
0812 09:49:24.954548 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260812 09:49:24.955698 13468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:24.947361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:41837 (local address 127.13.99.2:36177)
0812 09:49:24.948048 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:24.948090 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 09:49:24.949441 (+ 1351us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:49:24.950004 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:49:24.950011 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:49:24.950094 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:49:24.950599 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:49:24.950613 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:49:24.951440 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:49:24.951443 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:49:24.952362 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:49:24.952371 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:49:24.952887 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:49:24.952890 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:49:24.953119 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:24.953843 (+ 724us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 09:49:24.953848 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:49:24.953907 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 09:49:24.953995 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:49:24.954298 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:49:24.954308 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:49:24.954316 (+ 8us) client_negotiation.cc:769] Sending connection context
0812 09:49:24.954442 (+ 126us) client_negotiation.cc:240] Negotiation successful
0812 09:49:24.954730 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":122,"threads_started":1}
I20260812 09:49:24.956147 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786527905902985-13708-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786527905902985-13708-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786527905902985-13708-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786527905902985-13708-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.99.3:0
--local_ip_for_outbound_sockets=127.13.99.3
--webserver_interface=127.13.99.3
--webserver_port=0
--tserver_master_addrs=127.13.99.62:41837
--builtin_ntp_servers=127.13.99.20:43057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 09:49:24.956884 13462 heartbeater.cc:347] Connected to a master server at 127.13.99.62:41837
I20260812 09:49:24.957162 13462 heartbeater.cc:464] Registering TS with master...
I20260812 09:49:24.957686 13462 heartbeater.cc:511] Master 127.13.99.62:41837 requested a full tablet report, sending...
I20260812 09:49:24.958777 12910 ts_manager.cc:194] Registered new tserver with Master: da4f7e8e0df5467995d220b56b23ca48 (127.13.99.2:39901)
I20260812 09:49:24.959340 12910 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.99.2:36177
W20260812 09:49:25.072228 13469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:49:25.072463 13469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:49:25.072583 13469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:49:25.076565 13469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:49:25.076611 13469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:49:25.076694 13469 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.99.3
I20260812 09:49:25.081370 13469 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:43057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786527905902985-13708-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786527905902985-13708-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.99.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786527905902985-13708-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.99.3
--webserver_port=0
--tserver_master_addrs=127.13.99.62:41837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13469
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.99.3
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786527905902985-13708-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:49:25.082350 13469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:49:25.083662 13469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:49:25.093290 13475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:49:25.093331 13478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:49:25.093300 13476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:49:25.093614 13469 server_base.cc:1034] running on GCE node
I20260812 09:49:25.094002 13469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:49:25.094641 13469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:49:25.095841 13469 hybrid_clock.cc:648] HybridClock initialized: now 1786528165095825 us; error 31 us; skew 500 ppm
I20260812 09:49:25.097846 13469 webserver.cc:460] Webserver started at http://127.13.99.3:42281/ using document root <none> and password file <none>
I20260812 09:49:25.098430 13469 fs_manager.cc:359] Metadata directory not provided
I20260812 09:49:25.098533 13469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:49:25.098763 13469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:49:25.100837 13469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786527905902985-13708-0/minicluster-data/ts-2/data/instance:
uuid: "041b39af9b3748dda1d03f682b652de5"
format_stamp: "Formatted at 2026-08-12 09:49:25 on dist-test-slave-jq4t"
server_key: "0610c5bed2f0cf4f6c2d0112f8945774"
server_key_iv: "8cee038323d1d3ea45f371c99b642859"
server_key_version: "encryptionkey@0"
I20260812 09:49:25.101400 13469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786527905902985-13708-0/minicluster-data/ts-2/wal/instance:
uuid: "041b39af9b3748dda1d03f682b652de5"
format_stamp: "Formatted at 2026-08-12 09:49:25 on dist-test-slave-jq4t"
server_key: "0610c5bed2f0cf4f6c2d0112f8945774"
server_key_iv: "8cee038323d1d3ea45f371c99b642859"
server_key_version: "encryptionkey@0"
I20260812 09:49:25.105247 13469 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 09:49:25.107620 13492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:49:25.108892 13469 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 09:49:25.109037 13469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786527905902985-13708-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786527905902985-13708-0/minicluster-data/ts-2/wal
uuid: "041b39af9b3748dda1d03f682b652de5"
format_stamp: "Formatted at 2026-08-12 09:49:25 on dist-test-slave-jq4t"
server_key: "0610c5bed2f0cf4f6c2d0112f8945774"
server_key_iv: "8cee038323d1d3ea45f371c99b642859"
server_key_version: "encryptionkey@0"
I20260812 09:49:25.109146 13469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786527905902985-13708-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786527905902985-13708-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786527905902985-13708-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:49:25.122217 13469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:49:25.122823 13469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:49:25.123010 13469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:49:25.123613 13469 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 09:49:25.124581 13469 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:49:25.124650 13469 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 09:49:25.124717 13469 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:49:25.124764 13469 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 09:49:25.133877 13469 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.3:37721
I20260812 09:49:25.133948 13701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.3:37721 every 8 connection(s)
I20260812 09:49:25.134773 13469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786527905902985-13708-0/minicluster-data/ts-2/data/info.pb
I20260812 09:49:25.143428 12979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:25.136557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.3:54071 (local address 127.13.99.62:41837)
0812 09:49:25.136703 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:25.136707 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:25.137707 (+ 1000us) server_negotiation.cc:407] Connection header received
0812 09:49:25.139031 (+ 1324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:25.139033 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:25.139080 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:25.139154 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:49:25.139984 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:25.140563 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:25.141668 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:25.141882 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:25.141984 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:25.142982 (+ 998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:49:25.142985 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:49:25.142987 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 09:49:25.142998 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:49:25.143016 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 09:49:25.143024 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:49:25.143081 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:25.143265 (+ 184us) server_negotiation.cc:299] Negotiation successful
0812 09:49:25.143317 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260812 09:49:25.144001 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 13469
I20260812 09:49:25.144141 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786527905902985-13708-0/minicluster-data/ts-2/wal/instance
I20260812 09:49:25.144327 13706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:25.136947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:41837 (local address 127.13.99.3:54071)
0812 09:49:25.137563 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:25.137601 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0812 09:49:25.138834 (+ 1233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:49:25.139318 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:49:25.139324 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:49:25.139404 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:49:25.139858 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:49:25.139870 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:49:25.140674 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:49:25.140677 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:49:25.141554 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:49:25.141564 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:49:25.142038 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:49:25.142040 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:49:25.142177 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:25.142726 (+ 549us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 09:49:25.142728 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:49:25.142766 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 09:49:25.142825 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:49:25.143098 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:49:25.143105 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:49:25.143109 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 09:49:25.143202 (+ 93us) client_negotiation.cc:240] Negotiation successful
0812 09:49:25.143411 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":106,"threads_started":1}
I20260812 09:49:25.144472 13708 external_mini_cluster.cc:1428] Setting key 2c3aef94f8dae56546072b38d2be7d5e
I20260812 09:49:25.145552 13704 heartbeater.cc:347] Connected to a master server at 127.13.99.62:41837
I20260812 09:49:25.145813 13704 heartbeater.cc:464] Registering TS with master...
I20260812 09:49:25.146399 13704 heartbeater.cc:511] Master 127.13.99.62:41837 requested a full tablet report, sending...
I20260812 09:49:25.147415 12909 ts_manager.cc:194] Registered new tserver with Master: 041b39af9b3748dda1d03f682b652de5 (127.13.99.3:37721)
I20260812 09:49:25.147969 12909 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.99.3:54071
I20260812 09:49:25.158382 13708 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 09:49:25.164633 12979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:25.160566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54966 (local address 127.13.99.62:41837)
0812 09:49:25.160719 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:25.160724 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:25.160911 (+ 187us) server_negotiation.cc:407] Connection header received
0812 09:49:25.161061 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:25.161068 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:25.161107 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:25.161192 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:49:25.161494 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:25.162004 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:25.162864 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:25.163085 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:25.163196 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:25.164097 (+ 901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:49:25.164103 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:49:25.164110 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 09:49:25.164121 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:49:25.164139 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 09:49:25.164147 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:49:25.164283 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:25.164435 (+ 152us) server_negotiation.cc:299] Negotiation successful
0812 09:49:25.164490 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260812 09:49:25.167356 12909 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54966:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 09:49:25.170410 12909 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 09:49:25.184582 13725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:25.178646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.2:39901 (local address 127.0.0.1:51900)
0812 09:49:25.179439 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:25.179489 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0812 09:49:25.179619 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:49:25.180508 (+ 889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:49:25.180511 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:49:25.180531 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:49:25.180657 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:49:25.180667 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:49:25.182063 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:49:25.182066 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:49:25.183199 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:49:25.183210 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:49:25.184009 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:49:25.184012 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:49:25.184119 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:25.184177 (+ 58us) client_negotiation.cc:769] Sending connection context
0812 09:49:25.184289 (+ 112us) client_negotiation.cc:240] Negotiation successful
0812 09:49:25.184407 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"mutex_wait_us":38,"thread_start_us":80,"threads_started":1}
I20260812 09:49:25.184643 13728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:25.178767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51900 (local address 127.13.99.2:39901)
0812 09:49:25.180118 (+ 1351us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:25.180127 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:25.180147 (+ 20us) server_negotiation.cc:407] Connection header received
0812 09:49:25.180209 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:25.180213 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:25.180349 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:25.180539 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:49:25.180803 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:25.181935 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:25.183334 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:25.183899 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:25.184015 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:25.184202 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:25.184337 (+ 135us) server_negotiation.cc:299] Negotiation successful
0812 09:49:25.184435 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1253,"thread_start_us":109,"threads_started":1}
I20260812 09:49:25.184900 13729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:25.178755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.1:46507 (local address 127.0.0.1:55004)
0812 09:49:25.179587 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:25.179600 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 09:49:25.179691 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:49:25.180431 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:49:25.180438 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:49:25.180460 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:49:25.180619 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:49:25.180629 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:49:25.181784 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:49:25.181788 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:49:25.183139 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:49:25.183153 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:49:25.184461 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:49:25.184465 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:49:25.184577 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:25.184609 (+ 32us) client_negotiation.cc:769] Sending connection context
0812 09:49:25.184687 (+ 78us) client_negotiation.cc:240] Negotiation successful
0812 09:49:25.184765 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":765,"mutex_wait_us":1,"thread_start_us":104,"threads_started":1}
I20260812 09:49:25.185739 13727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:25.179188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.3:37721 (local address 127.0.0.1:49288)
0812 09:49:25.180711 (+ 1523us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:25.180726 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 09:49:25.180813 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:49:25.181317 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:49:25.181320 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:49:25.181339 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:49:25.181437 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:49:25.181442 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:49:25.182750 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:49:25.182753 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:49:25.183778 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:49:25.183789 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:49:25.185321 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:49:25.185325 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:49:25.185421 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:25.185446 (+ 25us) client_negotiation.cc:769] Sending connection context
0812 09:49:25.185526 (+ 80us) client_negotiation.cc:240] Negotiation successful
0812 09:49:25.185601 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1448,"mutex_wait_us":3,"thread_start_us":103,"threads_started":1}
I20260812 09:49:25.185832 13733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:25.179622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49288 (local address 127.13.99.3:37721)
0812 09:49:25.180014 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:25.180027 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:25.180920 (+ 893us) server_negotiation.cc:407] Connection header received
0812 09:49:25.180999 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:25.181004 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:25.181166 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:25.181310 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:49:25.181567 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:25.182626 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:25.183921 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:25.184748 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:25.185089 (+ 341us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:25.185255 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:25.185567 (+ 312us) server_negotiation.cc:299] Negotiation successful
0812 09:49:25.185665 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":185,"threads_started":1}
I20260812 09:49:25.185439 13731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:25.179182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55004 (local address 127.13.99.1:46507)
0812 09:49:25.179965 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:25.179979 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:25.180003 (+ 24us) server_negotiation.cc:407] Connection header received
0812 09:49:25.180086 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:25.180091 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:25.180262 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:25.180430 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:49:25.180759 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:25.181666 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:25.183290 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:25.184109 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:25.184211 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:25.184390 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:25.185148 (+ 758us) server_negotiation.cc:299] Negotiation successful
0812 09:49:25.185250 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":654,"thread_start_us":118,"threads_started":1}
I20260812 09:49:25.187949 13347 tablet_service.cc:1467] Processing CreateTablet for tablet 94f3ae2a7dc44bd7a514b71436144b33 (DEFAULT_TABLE table=test-table [id=457d0d40430b4f0bb4843db6b671cfef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 09:49:25.189159 13347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94f3ae2a7dc44bd7a514b71436144b33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:49:25.188894 13118 tablet_service.cc:1467] Processing CreateTablet for tablet 94f3ae2a7dc44bd7a514b71436144b33 (DEFAULT_TABLE table=test-table [id=457d0d40430b4f0bb4843db6b671cfef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 09:49:25.189903 13118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94f3ae2a7dc44bd7a514b71436144b33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:49:25.190867 13584 tablet_service.cc:1467] Processing CreateTablet for tablet 94f3ae2a7dc44bd7a514b71436144b33 (DEFAULT_TABLE table=test-table [id=457d0d40430b4f0bb4843db6b671cfef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 09:49:25.191582 13584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94f3ae2a7dc44bd7a514b71436144b33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:49:25.196825 13734 tablet_bootstrap.cc:492] T 94f3ae2a7dc44bd7a514b71436144b33 P fb2fb1cac5a94c5ebd6eb37dd31ef391: Bootstrap starting.
I20260812 09:49:25.197341 13737 tablet_bootstrap.cc:492] T 94f3ae2a7dc44bd7a514b71436144b33 P da4f7e8e0df5467995d220b56b23ca48: Bootstrap starting.
I20260812 09:49:25.197899 13740 tablet_bootstrap.cc:492] T 94f3ae2a7dc44bd7a514b71436144b33 P 041b39af9b3748dda1d03f682b652de5: Bootstrap starting.
I20260812 09:49:25.199479 13737 tablet_bootstrap.cc:654] T 94f3ae2a7dc44bd7a514b71436144b33 P da4f7e8e0df5467995d220b56b23ca48: Neither blocks nor log segments found. Creating new log.
I20260812 09:49:25.199647 13740 tablet_bootstrap.cc:654] T 94f3ae2a7dc44bd7a514b71436144b33 P 041b39af9b3748dda1d03f682b652de5: Neither blocks nor log segments found. Creating new log.
I20260812 09:49:25.200376 13737 log.cc:824] T 94f3ae2a7dc44bd7a514b71436144b33 P da4f7e8e0df5467995d220b56b23ca48: Log is configured to *not* fsync() on all Append() calls
I20260812 09:49:25.200539 13740 log.cc:824] T 94f3ae2a7dc44bd7a514b71436144b33 P 041b39af9b3748dda1d03f682b652de5: Log is configured to *not* fsync() on all Append() calls
I20260812 09:49:25.202454 13734 tablet_bootstrap.cc:654] T 94f3ae2a7dc44bd7a514b71436144b33 P fb2fb1cac5a94c5ebd6eb37dd31ef391: Neither blocks nor log segments found. Creating new log.
I20260812 09:49:25.202525 13740 tablet_bootstrap.cc:492] T 94f3ae2a7dc44bd7a514b71436144b33 P 041b39af9b3748dda1d03f682b652de5: No bootstrap required, opened a new log
I20260812 09:49:25.202528 13737 tablet_bootstrap.cc:492] T 94f3ae2a7dc44bd7a514b71436144b33 P da4f7e8e0df5467995d220b56b23ca48: No bootstrap required, opened a new log
I20260812 09:49:25.202721 13740 ts_tablet_manager.cc:1397] T 94f3ae2a7dc44bd7a514b71436144b33 P 041b39af9b3748dda1d03f682b652de5: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260812 09:49:25.202721 13737 ts_tablet_manager.cc:1397] T 94f3ae2a7dc44bd7a514b71436144b33 P da4f7e8e0df5467995d220b56b23ca48: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260812 09:49:25.203167 13734 log.cc:824] T 94f3ae2a7dc44bd7a514b71436144b33 P fb2fb1cac5a94c5ebd6eb37dd31ef391: Log is configured to *not* fsync() on all Append() calls
I20260812 09:49:25.205034 13734 tablet_bootstrap.cc:492] T 94f3ae2a7dc44bd7a514b71436144b33 P fb2fb1cac5a94c5ebd6eb37dd31ef391: No bootstrap required, opened a new log
I20260812 09:49:25.205240 13734 ts_tablet_manager.cc:1397] T 94f3ae2a7dc44bd7a514b71436144b33 P fb2fb1cac5a94c5ebd6eb37dd31ef391: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.004s
I20260812 09:49:25.207177 13737 raft_consensus.cc:348] T 94f3ae2a7dc44bd7a514b71436144b33 P da4f7e8e0df5467995d220b56b23ca48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da4f7e8e0df5467995d220b56b23ca48" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 39901 } } peers { permanent_uuid: "fb2fb1cac5a94c5ebd6eb37dd31ef391" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 46507 } } peers { permanent_uuid: "041b39af9b3748dda1d03f682b652de5" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 37721 } }
I20260812 09:49:25.207569 13737 raft_consensus.cc:374] T 94f3ae2a7dc44bd7a514b71436144b33 P da4f7e8e0df5467995d220b56b23ca48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:49:25.207448 13740 raft_consensus.cc:348] T 94f3ae2a7dc44bd7a514b71436144b33 P 041b39af9b3748dda1d03f682b652de5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da4f7e8e0df5467995d220b56b23ca48" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 39901 } } peers { permanent_uuid: "fb2fb1cac5a94c5ebd6eb37dd31ef391" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 46507 } } peers { permanent_uuid: "041b39af9b3748dda1d03f682b652de5" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 37721 } }
I20260812 09:49:25.207648 13737 raft_consensus.cc:729] T 94f3ae2a7dc44bd7a514b71436144b33 P da4f7e8e0df5467995d220b56b23ca48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da4f7e8e0df5467995d220b56b23ca48, State: Initialized, Role: FOLLOWER
I20260812 09:49:25.207692 13740 raft_consensus.cc:374] T 94f3ae2a7dc44bd7a514b71436144b33 P 041b39af9b3748dda1d03f682b652de5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:49:25.207779 13740 raft_consensus.cc:729] T 94f3ae2a7dc44bd7a514b71436144b33 P 041b39af9b3748dda1d03f682b652de5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 041b39af9b3748dda1d03f682b652de5, State: Initialized, Role: FOLLOWER
I20260812 09:49:25.208164 13737 consensus_queue.cc:260] T 94f3ae2a7dc44bd7a514b71436144b33 P da4f7e8e0df5467995d220b56b23ca48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da4f7e8e0df5467995d220b56b23ca48" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 39901 } } peers { permanent_uuid: "fb2fb1cac5a94c5ebd6eb37dd31ef391" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 46507 } } peers { permanent_uuid: "041b39af9b3748dda1d03f682b652de5" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 37721 } }
I20260812 09:49:25.208179 13740 consensus_queue.cc:260] T 94f3ae2a7dc44bd7a514b71436144b33 P 041b39af9b3748dda1d03f682b652de5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da4f7e8e0df5467995d220b56b23ca48" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 39901 } } peers { permanent_uuid: "fb2fb1cac5a94c5ebd6eb37dd31ef391" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 46507 } } peers { permanent_uuid: "041b39af9b3748dda1d03f682b652de5" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 37721 } }
I20260812 09:49:25.209121 13462 heartbeater.cc:503] Master 127.13.99.62:41837 was elected leader, sending a full tablet report...
I20260812 09:49:25.209127 13704 heartbeater.cc:503] Master 127.13.99.62:41837 was elected leader, sending a full tablet report...
I20260812 09:49:25.209231 13740 ts_tablet_manager.cc:1428] T 94f3ae2a7dc44bd7a514b71436144b33 P 041b39af9b3748dda1d03f682b652de5: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260812 09:49:25.209231 13737 ts_tablet_manager.cc:1428] T 94f3ae2a7dc44bd7a514b71436144b33 P da4f7e8e0df5467995d220b56b23ca48: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260812 09:49:25.210315 13734 raft_consensus.cc:348] T 94f3ae2a7dc44bd7a514b71436144b33 P fb2fb1cac5a94c5ebd6eb37dd31ef391 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da4f7e8e0df5467995d220b56b23ca48" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 39901 } } peers { permanent_uuid: "fb2fb1cac5a94c5ebd6eb37dd31ef391" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 46507 } } peers { permanent_uuid: "041b39af9b3748dda1d03f682b652de5" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 37721 } }
I20260812 09:49:25.210580 13734 raft_consensus.cc:374] T 94f3ae2a7dc44bd7a514b71436144b33 P fb2fb1cac5a94c5ebd6eb37dd31ef391 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:49:25.210644 13734 raft_consensus.cc:729] T 94f3ae2a7dc44bd7a514b71436144b33 P fb2fb1cac5a94c5ebd6eb37dd31ef391 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb2fb1cac5a94c5ebd6eb37dd31ef391, State: Initialized, Role: FOLLOWER
I20260812 09:49:25.211084 13734 consensus_queue.cc:260] T 94f3ae2a7dc44bd7a514b71436144b33 P fb2fb1cac5a94c5ebd6eb37dd31ef391 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da4f7e8e0df5467995d220b56b23ca48" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 39901 } } peers { permanent_uuid: "fb2fb1cac5a94c5ebd6eb37dd31ef391" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 46507 } } peers { permanent_uuid: "041b39af9b3748dda1d03f682b652de5" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 37721 } }
I20260812 09:49:25.211717 13218 heartbeater.cc:503] Master 127.13.99.62:41837 was elected leader, sending a full tablet report...
I20260812 09:49:25.212108 13734 ts_tablet_manager.cc:1428] T 94f3ae2a7dc44bd7a514b71436144b33 P fb2fb1cac5a94c5ebd6eb37dd31ef391: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
W20260812 09:49:25.255585 13219 tablet.cc:2369] T 94f3ae2a7dc44bd7a514b71436144b33 P fb2fb1cac5a94c5ebd6eb37dd31ef391: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 09:49:25.348874 13747 raft_consensus.cc:482] T 94f3ae2a7dc44bd7a514b71436144b33 P da4f7e8e0df5467995d220b56b23ca48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 09:49:25.349095 13747 raft_consensus.cc:504] T 94f3ae2a7dc44bd7a514b71436144b33 P da4f7e8e0df5467995d220b56b23ca48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da4f7e8e0df5467995d220b56b23ca48" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 39901 } } peers { permanent_uuid: "fb2fb1cac5a94c5ebd6eb37dd31ef391" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 46507 } } peers { permanent_uuid: "041b39af9b3748dda1d03f682b652de5" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 37721 } }
I20260812 09:49:25.350245 13747 leader_election.cc:288] T 94f3ae2a7dc44bd7a514b71436144b33 P da4f7e8e0df5467995d220b56b23ca48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb2fb1cac5a94c5ebd6eb37dd31ef391 (127.13.99.1:46507), 041b39af9b3748dda1d03f682b652de5 (127.13.99.3:37721)
I20260812 09:49:25.355530 13731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:25.350638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.2:44643 (local address 127.13.99.1:46507)
0812 09:49:25.350782 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:25.350787 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:25.350824 (+ 37us) server_negotiation.cc:407] Connection header received
0812 09:49:25.351073 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:25.351097 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:25.351161 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:25.351240 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:49:25.351623 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:25.352413 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:25.354187 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:25.354797 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:25.354916 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:25.354982 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:25.355326 (+ 344us) server_negotiation.cc:299] Negotiation successful
0812 09:49:25.355410 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260812 09:49:25.355530 13733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:25.350687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.2:41955 (local address 127.13.99.3:37721)
0812 09:49:25.350845 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:25.350849 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:25.351131 (+ 282us) server_negotiation.cc:407] Connection header received
0812 09:49:25.351198 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:25.351203 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:25.351253 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:25.351322 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:49:25.351614 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:25.352407 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:25.354187 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:25.354797 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:25.354933 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:25.354982 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:25.355326 (+ 344us) server_negotiation.cc:299] Negotiation successful
0812 09:49:25.355396 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 09:49:25.355553 13468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:25.350523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.1:46507 (local address 127.13.99.2:44643)
0812 09:49:25.350750 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:25.350766 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 09:49:25.350910 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:49:25.351276 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:49:25.351278 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:49:25.351314 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:49:25.351474 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:49:25.351481 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:49:25.352533 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:49:25.352536 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:49:25.354030 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:49:25.354042 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:49:25.354930 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:49:25.354934 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:49:25.355168 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:25.355205 (+ 37us) client_negotiation.cc:769] Sending connection context
0812 09:49:25.355289 (+ 84us) client_negotiation.cc:240] Negotiation successful
0812 09:49:25.355397 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"mutex_wait_us":214}
I20260812 09:49:25.355562 13750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:25.350542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.3:37721 (local address 127.13.99.2:41955)
0812 09:49:25.351021 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:25.351032 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 09:49:25.351122 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:49:25.351352 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:49:25.351355 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:49:25.351368 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:49:25.351478 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:49:25.351483 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:49:25.352533 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:49:25.352536 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:49:25.354037 (+ 1501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:49:25.354043 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:49:25.354942 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:49:25.354945 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:49:25.355163 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:25.355214 (+ 51us) client_negotiation.cc:769] Sending connection context
0812 09:49:25.355289 (+ 75us) client_negotiation.cc:240] Negotiation successful
0812 09:49:25.355397 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"mutex_wait_us":216,"thread_start_us":147,"threads_started":1}
I20260812 09:49:25.356282 13157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94f3ae2a7dc44bd7a514b71436144b33" candidate_uuid: "da4f7e8e0df5467995d220b56b23ca48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb2fb1cac5a94c5ebd6eb37dd31ef391" is_pre_election: true
I20260812 09:49:25.356293 13621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94f3ae2a7dc44bd7a514b71436144b33" candidate_uuid: "da4f7e8e0df5467995d220b56b23ca48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "041b39af9b3748dda1d03f682b652de5" is_pre_election: true
I20260812 09:49:25.356617 13621 raft_consensus.cc:2454] T 94f3ae2a7dc44bd7a514b71436144b33 P 041b39af9b3748dda1d03f682b652de5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da4f7e8e0df5467995d220b56b23ca48 in term 0.
I20260812 09:49:25.356618 13157 raft_consensus.cc:2454] T 94f3ae2a7dc44bd7a514b71436144b33 P fb2fb1cac5a94c5ebd6eb37dd31ef391 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da4f7e8e0df5467995d220b56b23ca48 in term 0.
I20260812 09:49:25.357087 13255 leader_election.cc:302] T 94f3ae2a7dc44bd7a514b71436144b33 P da4f7e8e0df5467995d220b56b23ca48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 041b39af9b3748dda1d03f682b652de5, da4f7e8e0df5467995d220b56b23ca48; no voters:
I20260812 09:49:25.357364 13747 raft_consensus.cc:2781] T 94f3ae2a7dc44bd7a514b71436144b33 P da4f7e8e0df5467995d220b56b23ca48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 09:49:25.357462 13747 raft_consensus.cc:482] T 94f3ae2a7dc44bd7a514b71436144b33 P da4f7e8e0df5467995d220b56b23ca48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 09:49:25.357527 13747 raft_consensus.cc:3037] T 94f3ae2a7dc44bd7a514b71436144b33 P da4f7e8e0df5467995d220b56b23ca48 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:49:25.358556 13747 raft_consensus.cc:504] T 94f3ae2a7dc44bd7a514b71436144b33 P da4f7e8e0df5467995d220b56b23ca48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da4f7e8e0df5467995d220b56b23ca48" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 39901 } } peers { permanent_uuid: "fb2fb1cac5a94c5ebd6eb37dd31ef391" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 46507 } } peers { permanent_uuid: "041b39af9b3748dda1d03f682b652de5" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 37721 } }
I20260812 09:49:25.358922 13747 leader_election.cc:288] T 94f3ae2a7dc44bd7a514b71436144b33 P da4f7e8e0df5467995d220b56b23ca48 [CANDIDATE]: Term 1 election: Requested vote from peers fb2fb1cac5a94c5ebd6eb37dd31ef391 (127.13.99.1:46507), 041b39af9b3748dda1d03f682b652de5 (127.13.99.3:37721)
I20260812 09:49:25.359248 13157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94f3ae2a7dc44bd7a514b71436144b33" candidate_uuid: "da4f7e8e0df5467995d220b56b23ca48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb2fb1cac5a94c5ebd6eb37dd31ef391"
I20260812 09:49:25.359318 13621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94f3ae2a7dc44bd7a514b71436144b33" candidate_uuid: "da4f7e8e0df5467995d220b56b23ca48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "041b39af9b3748dda1d03f682b652de5"
I20260812 09:49:25.359388 13157 raft_consensus.cc:3037] T 94f3ae2a7dc44bd7a514b71436144b33 P fb2fb1cac5a94c5ebd6eb37dd31ef391 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:49:25.359455 13621 raft_consensus.cc:3037] T 94f3ae2a7dc44bd7a514b71436144b33 P 041b39af9b3748dda1d03f682b652de5 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:49:25.360598 13157 raft_consensus.cc:2454] T 94f3ae2a7dc44bd7a514b71436144b33 P fb2fb1cac5a94c5ebd6eb37dd31ef391 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da4f7e8e0df5467995d220b56b23ca48 in term 1.
I20260812 09:49:25.360683 13621 raft_consensus.cc:2454] T 94f3ae2a7dc44bd7a514b71436144b33 P 041b39af9b3748dda1d03f682b652de5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da4f7e8e0df5467995d220b56b23ca48 in term 1.
I20260812 09:49:25.360929 13256 leader_election.cc:302] T 94f3ae2a7dc44bd7a514b71436144b33 P da4f7e8e0df5467995d220b56b23ca48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: da4f7e8e0df5467995d220b56b23ca48, fb2fb1cac5a94c5ebd6eb37dd31ef391; no voters:
I20260812 09:49:25.361114 13747 raft_consensus.cc:2781] T 94f3ae2a7dc44bd7a514b71436144b33 P da4f7e8e0df5467995d220b56b23ca48 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 09:49:25.361336 13747 raft_consensus.cc:686] T 94f3ae2a7dc44bd7a514b71436144b33 P da4f7e8e0df5467995d220b56b23ca48 [term 1 LEADER]: Becoming Leader. State: Replica: da4f7e8e0df5467995d220b56b23ca48, State: Running, Role: LEADER
I20260812 09:49:25.361655 13747 consensus_queue.cc:237] T 94f3ae2a7dc44bd7a514b71436144b33 P da4f7e8e0df5467995d220b56b23ca48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da4f7e8e0df5467995d220b56b23ca48" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 39901 } } peers { permanent_uuid: "fb2fb1cac5a94c5ebd6eb37dd31ef391" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 46507 } } peers { permanent_uuid: "041b39af9b3748dda1d03f682b652de5" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 37721 } }
I20260812 09:49:25.364884 12910 catalog_manager.cc:5495] T 94f3ae2a7dc44bd7a514b71436144b33 P da4f7e8e0df5467995d220b56b23ca48 reported cstate change: term changed from 0 to 1, leader changed from <none> to da4f7e8e0df5467995d220b56b23ca48 (127.13.99.2). New cstate: current_term: 1 leader_uuid: "da4f7e8e0df5467995d220b56b23ca48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da4f7e8e0df5467995d220b56b23ca48" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 39901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fb2fb1cac5a94c5ebd6eb37dd31ef391" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "041b39af9b3748dda1d03f682b652de5" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 37721 } health_report { overall_health: UNKNOWN } } }
W20260812 09:49:25.385725 13705 tablet.cc:2369] T 94f3ae2a7dc44bd7a514b71436144b33 P 041b39af9b3748dda1d03f682b652de5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 09:49:25.419697 13728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:25.416026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60828 (local address 127.13.99.2:39901)
0812 09:49:25.416182 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:25.416186 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:25.416203 (+ 17us) server_negotiation.cc:407] Connection header received
0812 09:49:25.416328 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:25.416331 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:25.416399 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:25.416547 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 09:49:25.416807 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:25.417325 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:25.418192 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:25.418428 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:25.418541 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:25.418849 (+ 308us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 09:49:25.419255 (+ 406us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 09:49:25.419417 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:25.419511 (+ 94us) server_negotiation.cc:299] Negotiation successful
0812 09:49:25.419546 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 09:49:25.426020 13621 raft_consensus.cc:1261] T 94f3ae2a7dc44bd7a514b71436144b33 P 041b39af9b3748dda1d03f682b652de5 [term 1 FOLLOWER]: Refusing update from remote peer da4f7e8e0df5467995d220b56b23ca48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 09:49:25.426052 13157 raft_consensus.cc:1261] T 94f3ae2a7dc44bd7a514b71436144b33 P fb2fb1cac5a94c5ebd6eb37dd31ef391 [term 1 FOLLOWER]: Refusing update from remote peer da4f7e8e0df5467995d220b56b23ca48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 09:49:25.426726 13747 consensus_queue.cc:1035] T 94f3ae2a7dc44bd7a514b71436144b33 P da4f7e8e0df5467995d220b56b23ca48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "041b39af9b3748dda1d03f682b652de5" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 37721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 09:49:25.426905 13751 consensus_queue.cc:1035] T 94f3ae2a7dc44bd7a514b71436144b33 P da4f7e8e0df5467995d220b56b23ca48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb2fb1cac5a94c5ebd6eb37dd31ef391" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 09:49:25.435233 13757 mvcc.cc:205] Tried to move back new op lower bound from 7317619365576548352 to 7317619365323943936. Current Snapshot: MvccSnapshot[applied={T|T < 7317619365576548352}]
I20260812 09:49:25.435233 13758 mvcc.cc:205] Tried to move back new op lower bound from 7317619365576548352 to 7317619365323943936. Current Snapshot: MvccSnapshot[applied={T|T < 7317619365576548352}]
I20260812 09:49:25.443452 12909 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54966:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 09:49:25.443624 12909 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54966:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 09:49:25.445884 12909 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3fc106fe4c55423597446f25e9db4863: Sending DeleteTablet for 3 replicas of tablet 94f3ae2a7dc44bd7a514b71436144b33
I20260812 09:49:25.446659 13347 tablet_service.cc:1514] Processing DeleteTablet for tablet 94f3ae2a7dc44bd7a514b71436144b33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 09:49:25 UTC) from {username='slave'} at 127.0.0.1:51900
I20260812 09:49:25.446832 13118 tablet_service.cc:1514] Processing DeleteTablet for tablet 94f3ae2a7dc44bd7a514b71436144b33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 09:49:25 UTC) from {username='slave'} at 127.0.0.1:55004
I20260812 09:49:25.446938 13584 tablet_service.cc:1514] Processing DeleteTablet for tablet 94f3ae2a7dc44bd7a514b71436144b33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 09:49:25 UTC) from {username='slave'} at 127.0.0.1:49288
I20260812 09:49:25.447288 13777 tablet_replica.cc:330] T 94f3ae2a7dc44bd7a514b71436144b33 P fb2fb1cac5a94c5ebd6eb37dd31ef391: stopping tablet replica
I20260812 09:49:25.447291 13776 tablet_replica.cc:330] T 94f3ae2a7dc44bd7a514b71436144b33 P da4f7e8e0df5467995d220b56b23ca48: stopping tablet replica
I20260812 09:49:25.447593 13776 raft_consensus.cc:2229] T 94f3ae2a7dc44bd7a514b71436144b33 P da4f7e8e0df5467995d220b56b23ca48 [term 1 LEADER]: Raft consensus shutting down.
I20260812 09:49:25.447636 13777 raft_consensus.cc:2229] T 94f3ae2a7dc44bd7a514b71436144b33 P fb2fb1cac5a94c5ebd6eb37dd31ef391 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 09:49:25.447929 13776 raft_consensus.cc:2258] T 94f3ae2a7dc44bd7a514b71436144b33 P da4f7e8e0df5467995d220b56b23ca48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 09:49:25.448120 13777 raft_consensus.cc:2258] T 94f3ae2a7dc44bd7a514b71436144b33 P fb2fb1cac5a94c5ebd6eb37dd31ef391 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 09:49:25.448184 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 13015
I20260812 09:49:25.449608 13776 ts_tablet_manager.cc:1905] T 94f3ae2a7dc44bd7a514b71436144b33 P da4f7e8e0df5467995d220b56b23ca48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 09:49:25.452270 13776 ts_tablet_manager.cc:1918] T 94f3ae2a7dc44bd7a514b71436144b33 P da4f7e8e0df5467995d220b56b23ca48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 09:49:25.452389 13776 log.cc:1196] T 94f3ae2a7dc44bd7a514b71436144b33 P da4f7e8e0df5467995d220b56b23ca48: Deleting WAL directory at /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786527905902985-13708-0/minicluster-data/ts-1/wal/wals/94f3ae2a7dc44bd7a514b71436144b33
I20260812 09:49:25.452765 13776 ts_tablet_manager.cc:1939] T 94f3ae2a7dc44bd7a514b71436144b33 P da4f7e8e0df5467995d220b56b23ca48: Deleting consensus metadata
I20260812 09:49:25.454051 13779 tablet_replica.cc:330] T 94f3ae2a7dc44bd7a514b71436144b33 P 041b39af9b3748dda1d03f682b652de5: stopping tablet replica
I20260812 09:49:25.454058 12888 catalog_manager.cc:4841] TS da4f7e8e0df5467995d220b56b23ca48 (127.13.99.2:39901): tablet 94f3ae2a7dc44bd7a514b71436144b33 (table test-table [id=457d0d40430b4f0bb4843db6b671cfef]) successfully deleted
I20260812 09:49:25.454388 13779 raft_consensus.cc:2229] T 94f3ae2a7dc44bd7a514b71436144b33 P 041b39af9b3748dda1d03f682b652de5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 09:49:25.454715 13779 raft_consensus.cc:2258] T 94f3ae2a7dc44bd7a514b71436144b33 P 041b39af9b3748dda1d03f682b652de5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 09:49:25.456238 13779 ts_tablet_manager.cc:1905] T 94f3ae2a7dc44bd7a514b71436144b33 P 041b39af9b3748dda1d03f682b652de5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 09:49:25.458384 13779 ts_tablet_manager.cc:1918] T 94f3ae2a7dc44bd7a514b71436144b33 P 041b39af9b3748dda1d03f682b652de5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 09:49:25.458508 13779 log.cc:1196] T 94f3ae2a7dc44bd7a514b71436144b33 P 041b39af9b3748dda1d03f682b652de5: Deleting WAL directory at /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786527905902985-13708-0/minicluster-data/ts-2/wal/wals/94f3ae2a7dc44bd7a514b71436144b33
I20260812 09:49:25.458820 13779 ts_tablet_manager.cc:1939] T 94f3ae2a7dc44bd7a514b71436144b33 P 041b39af9b3748dda1d03f682b652de5: Deleting consensus metadata
I20260812 09:49:25.459863 12883 catalog_manager.cc:4841] TS 041b39af9b3748dda1d03f682b652de5 (127.13.99.3:37721): tablet 94f3ae2a7dc44bd7a514b71436144b33 (table test-table [id=457d0d40430b4f0bb4843db6b671cfef]) successfully deleted
W20260812 09:49:25.463548 12887 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.99.1:46507 (error 108)
I20260812 09:49:25.463629 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 13221
I20260812 09:49:25.473853 13729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:25.473557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.1:46507 (local address 127.0.0.1:39202)
0812 09:49:25.473701 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:25.473778 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.1:46507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260812 09:49:25.474064 12887 catalog_manager.cc:4584] TS fb2fb1cac5a94c5ebd6eb37dd31ef391 (127.13.99.1:46507): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 94f3ae2a7dc44bd7a514b71436144b33: Network error: Client connection negotiation failed: client connection to 127.13.99.1:46507: connect: Connection refused (error 111)
I20260812 09:49:25.478747 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 13469
I20260812 09:49:25.488569 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 12858
2026-08-12T09:49:25Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1191 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 09:49:25 dist-test-slave-jq4t krb5kdc[13788](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 09:49:25 dist-test-slave-jq4t krb5kdc[13788](info): set up 2 sockets
Aug 12 09:49:25 dist-test-slave-jq4t krb5kdc[13788](info): commencing operation
krb5kdc: starting...
W20260812 09:49:26.874472 13708 mini_kdc.cc:121] Time spent starting KDC: real 1.360s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/krb5kdc/test-user.keytab.
Aug 12 09:49:26 dist-test-slave-jq4t krb5kdc[13788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T09:49:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T09:49:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.99.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.99.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.99.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab.
Entry for principal kudu/127.13.99.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab.
I20260812 09:49:27.019824 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.62:46849
--webserver_interface=127.13.99.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab
--principal=kudu/127.13.99.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:45893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.99.62:46849
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:49:27.136574 13810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:49:27.136826 13810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:49:27.136873 13810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:49:27.140906 13810 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 09:49:27.140960 13810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:49:27.140996 13810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:49:27.141053 13810 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 09:49:27.141088 13810 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 09:49:27.146270 13810 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:45893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.99.62:46849
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab
--principal=kudu/127.13.99.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.62:46849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.99.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13810
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:49:27.147289 13810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:49:27.148329 13810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:49:27.156965 13818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:49:27.156960 13822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:49:27.156960 13819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:49:27.157218 13810 server_base.cc:1034] running on GCE node
I20260812 09:49:27.157866 13810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:49:27.158957 13810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:49:27.160117 13810 hybrid_clock.cc:648] HybridClock initialized: now 1786528167160081 us; error 47 us; skew 500 ppm
Aug 12 09:49:27 dist-test-slave-jq4t krb5kdc[13788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528167, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:49:27.163002 13810 init.cc:377] Logged in from keytab as kudu/127.13.99.62@KRBTEST.COM (short username kudu)
I20260812 09:49:27.163921 13810 webserver.cc:460] Webserver started at http://127.13.99.62:37475/ using document root <none> and password file <none>
I20260812 09:49:27.164552 13810 fs_manager.cc:359] Metadata directory not provided
I20260812 09:49:27.164619 13810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:49:27.164855 13810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:49:27.166878 13810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/minicluster-data/master-0/data/instance:
uuid: "7fa7fc0aaebe44cfa7ee046a36462387"
format_stamp: "Formatted at 2026-08-12 09:49:27 on dist-test-slave-jq4t"
server_key: "15c9210a274c80545bba8065e1b76a8d"
server_key_iv: "d0b244d1337cee8d08c13084164543c5"
server_key_version: "encryptionkey@0"
I20260812 09:49:27.167460 13810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/minicluster-data/master-0/wal/instance:
uuid: "7fa7fc0aaebe44cfa7ee046a36462387"
format_stamp: "Formatted at 2026-08-12 09:49:27 on dist-test-slave-jq4t"
server_key: "15c9210a274c80545bba8065e1b76a8d"
server_key_iv: "d0b244d1337cee8d08c13084164543c5"
server_key_version: "encryptionkey@0"
I20260812 09:49:27.171197 13810 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260812 09:49:27.173873 13835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:49:27.175104 13810 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260812 09:49:27.175249 13810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/minicluster-data/master-0/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/minicluster-data/master-0/wal
uuid: "7fa7fc0aaebe44cfa7ee046a36462387"
format_stamp: "Formatted at 2026-08-12 09:49:27 on dist-test-slave-jq4t"
server_key: "15c9210a274c80545bba8065e1b76a8d"
server_key_iv: "d0b244d1337cee8d08c13084164543c5"
server_key_version: "encryptionkey@0"
I20260812 09:49:27.175364 13810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:49:27.194629 13810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:49:27.197566 13810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:49:27.197770 13810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:49:27.205134 13810 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.62:46849
I20260812 09:49:27.205163 13930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.62:46849 every 8 connection(s)
I20260812 09:49:27.206099 13810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
I20260812 09:49:27.208166 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 13810
I20260812 09:49:27.208293 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/minicluster-data/master-0/wal/instance
I20260812 09:49:27.208662 13708 external_mini_cluster.cc:1428] Setting key 3fe30b200d66aa7e7190aa4fcb9d40a7
I20260812 09:49:27.209302 13931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:49:27.215752 13931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fa7fc0aaebe44cfa7ee046a36462387: Bootstrap starting.
Aug 12 09:49:27 dist-test-slave-jq4t krb5kdc[13788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
I20260812 09:49:27.218459 13931 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7fa7fc0aaebe44cfa7ee046a36462387: Neither blocks nor log segments found. Creating new log.
I20260812 09:49:27.219342 13931 log.cc:824] T 00000000000000000000000000000000 P 7fa7fc0aaebe44cfa7ee046a36462387: Log is configured to *not* fsync() on all Append() calls
I20260812 09:49:27.221421 13931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fa7fc0aaebe44cfa7ee046a36462387: No bootstrap required, opened a new log
I20260812 09:49:27.223146 13937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:27.210157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51416 (local address 127.13.99.62:46849)
0812 09:49:27.210619 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:27.210635 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:27.210672 (+ 37us) server_negotiation.cc:407] Connection header received
0812 09:49:27.211391 (+ 719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:27.211412 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:27.211732 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:27.212083 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:49:27.213021 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:27.214109 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:27.215092 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:27.215566 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:27.215735 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:27.218598 (+ 2863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:49:27.218606 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:49:27.218630 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:49:27.218658 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:49:27.220312 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:49:27.220797 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:49:27.220803 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:49:27.220808 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:49:27.220923 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:49:27.221224 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:49:27.221227 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:49:27.221229 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:49:27.221681 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:49:27.221883 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:27.222177 (+ 294us) server_negotiation.cc:299] Negotiation successful
0812 09:49:27.222455 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":156,"threads_started":1}
I20260812 09:49:27.225167 13931 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7fa7fc0aaebe44cfa7ee046a36462387 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fa7fc0aaebe44cfa7ee046a36462387" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 46849 } }
I20260812 09:49:27.225456 13931 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7fa7fc0aaebe44cfa7ee046a36462387 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:49:27.225543 13931 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7fa7fc0aaebe44cfa7ee046a36462387 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa7fc0aaebe44cfa7ee046a36462387, State: Initialized, Role: FOLLOWER
I20260812 09:49:27.225953 13931 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7fa7fc0aaebe44cfa7ee046a36462387 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fa7fc0aaebe44cfa7ee046a36462387" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 46849 } }
I20260812 09:49:27.226100 13931 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7fa7fc0aaebe44cfa7ee046a36462387 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 09:49:27.226169 13931 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7fa7fc0aaebe44cfa7ee046a36462387 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 09:49:27.226279 13931 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7fa7fc0aaebe44cfa7ee046a36462387 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:49:27.227186 13931 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7fa7fc0aaebe44cfa7ee046a36462387 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fa7fc0aaebe44cfa7ee046a36462387" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 46849 } }
I20260812 09:49:27.227564 13931 leader_election.cc:302] T 00000000000000000000000000000000 P 7fa7fc0aaebe44cfa7ee046a36462387 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7fa7fc0aaebe44cfa7ee046a36462387; no voters:
I20260812 09:49:27.227921 13931 leader_election.cc:288] T 00000000000000000000000000000000 P 7fa7fc0aaebe44cfa7ee046a36462387 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 09:49:27.228046 13940 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7fa7fc0aaebe44cfa7ee046a36462387 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 09:49:27.228276 13940 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7fa7fc0aaebe44cfa7ee046a36462387 [term 1 LEADER]: Becoming Leader. State: Replica: 7fa7fc0aaebe44cfa7ee046a36462387, State: Running, Role: LEADER
I20260812 09:49:27.228621 13940 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7fa7fc0aaebe44cfa7ee046a36462387 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fa7fc0aaebe44cfa7ee046a36462387" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 46849 } }
I20260812 09:49:27.229180 13931 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7fa7fc0aaebe44cfa7ee046a36462387 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 09:49:27.230216 13942 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fa7fc0aaebe44cfa7ee046a36462387 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7fa7fc0aaebe44cfa7ee046a36462387. Latest consensus state: current_term: 1 leader_uuid: "7fa7fc0aaebe44cfa7ee046a36462387" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fa7fc0aaebe44cfa7ee046a36462387" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 46849 } } }
I20260812 09:49:27.231109 13942 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fa7fc0aaebe44cfa7ee046a36462387 [sys.catalog]: This master's current role is: LEADER
I20260812 09:49:27.230966 13941 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fa7fc0aaebe44cfa7ee046a36462387 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7fa7fc0aaebe44cfa7ee046a36462387" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fa7fc0aaebe44cfa7ee046a36462387" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 46849 } } }
I20260812 09:49:27.231459 13941 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fa7fc0aaebe44cfa7ee046a36462387 [sys.catalog]: This master's current role is: LEADER
W20260812 09:49:27.233901 13964 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7fa7fc0aaebe44cfa7ee046a36462387: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 09:49:27.233991 13964 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 09:49:27.234076 13962 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 09:49:27.234802 13962 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 09:49:27.240067 13962 catalog_manager.cc:1347] Generated new cluster ID: ef38e09e7df84d1889ecfd1b456e87f0
I20260812 09:49:27.240129 13962 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 09:49:27.266361 13962 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 09:49:27.267711 13962 catalog_manager.cc:1504] Loading token signing keys...
I20260812 09:49:27.277269 13962 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7fa7fc0aaebe44cfa7ee046a36462387: Generated new TSK 0
I20260812 09:49:27.278123 13962 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.99.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.99.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.99.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.1.keytab.
Entry for principal kudu/127.13.99.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.1.keytab.
I20260812 09:49:27.311112 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.99.1:0
--local_ip_for_outbound_sockets=127.13.99.1
--webserver_interface=127.13.99.1
--webserver_port=0
--tserver_master_addrs=127.13.99.62:46849
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.1.keytab
--principal=kudu/127.13.99.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:45893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:49:27.426661 13970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:49:27.426941 13970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:49:27.427018 13970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:49:27.431396 13970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:49:27.431471 13970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:49:27.431565 13970 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.99.1
I20260812 09:49:27.436743 13970 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:45893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.1.keytab
--principal=kudu/127.13.99.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.99.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.99.1
--webserver_port=0
--tserver_master_addrs=127.13.99.62:46849
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13970
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.99.1
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:49:27.437731 13970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:49:27.438735 13970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:49:27.448360 13980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:49:27.448354 13977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:49:27.448395 13976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:49:27.448733 13970 server_base.cc:1034] running on GCE node
I20260812 09:49:27.449138 13970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:49:27.449774 13970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:49:27.450963 13970 hybrid_clock.cc:648] HybridClock initialized: now 1786528167450937 us; error 36 us; skew 500 ppm
Aug 12 09:49:27 dist-test-slave-jq4t krb5kdc[13788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528167, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:49:27.453882 13970 init.cc:377] Logged in from keytab as kudu/127.13.99.1@KRBTEST.COM (short username kudu)
I20260812 09:49:27.454720 13970 webserver.cc:460] Webserver started at http://127.13.99.1:43205/ using document root <none> and password file <none>
I20260812 09:49:27.455251 13970 fs_manager.cc:359] Metadata directory not provided
I20260812 09:49:27.455309 13970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:49:27.455471 13970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:49:27.457484 13970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/minicluster-data/ts-0/data/instance:
uuid: "ce0c794ffd2c4812b9d591ffe17aeb6f"
format_stamp: "Formatted at 2026-08-12 09:49:27 on dist-test-slave-jq4t"
server_key: "51fde09826c6e09c008cb70e79cbf4fe"
server_key_iv: "17a567c4fa1b05c2927e536bf5661136"
server_key_version: "encryptionkey@0"
I20260812 09:49:27.458063 13970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/minicluster-data/ts-0/wal/instance:
uuid: "ce0c794ffd2c4812b9d591ffe17aeb6f"
format_stamp: "Formatted at 2026-08-12 09:49:27 on dist-test-slave-jq4t"
server_key: "51fde09826c6e09c008cb70e79cbf4fe"
server_key_iv: "17a567c4fa1b05c2927e536bf5661136"
server_key_version: "encryptionkey@0"
I20260812 09:49:27.461654 13970 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 09:49:27.464000 13996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:49:27.465342 13970 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 09:49:27.465482 13970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/minicluster-data/ts-0/wal
uuid: "ce0c794ffd2c4812b9d591ffe17aeb6f"
format_stamp: "Formatted at 2026-08-12 09:49:27 on dist-test-slave-jq4t"
server_key: "51fde09826c6e09c008cb70e79cbf4fe"
server_key_iv: "17a567c4fa1b05c2927e536bf5661136"
server_key_version: "encryptionkey@0"
I20260812 09:49:27.465591 13970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:49:27.482851 13970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:49:27.486177 13970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:49:27.486385 13970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:49:27.486982 13970 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 09:49:27.487821 13970 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:49:27.487900 13970 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 09:49:27.487972 13970 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:49:27.488026 13970 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 09:49:27.497108 13970 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.1:36301
I20260812 09:49:27.497136 14171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.1:36301 every 8 connection(s)
I20260812 09:49:27.498003 13970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/minicluster-data/ts-0/data/info.pb
I20260812 09:49:27.498512 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 13970
I20260812 09:49:27.498646 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/minicluster-data/ts-0/wal/instance
I20260812 09:49:27.498950 13708 external_mini_cluster.cc:1428] Setting key 7bd7cab20ceccab62aa69d2453e1ded4
Aug 12 09:49:27 dist-test-slave-jq4t krb5kdc[13788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528167, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.1@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.99.2@KRBTEST.COM; defaulting to no policy
I20260812 09:49:27.513175 13937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:27.499806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.1:44775 (local address 127.13.99.62:46849)
0812 09:49:27.499947 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:27.499951 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:27.501337 (+ 1386us) server_negotiation.cc:407] Connection header received
0812 09:49:27.502645 (+ 1308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:27.502680 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:27.502750 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:27.502860 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:49:27.504018 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:27.504701 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:27.505831 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:27.506073 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:27.506192 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:27.509181 (+ 2989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:49:27.509186 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:49:27.509189 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:49:27.509220 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:49:27.510971 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:49:27.511544 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:49:27.511548 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:49:27.511550 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:49:27.511640 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:49:27.512004 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:49:27.512008 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:49:27.512010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:49:27.512201 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:49:27.512316 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:27.512866 (+ 550us) server_negotiation.cc:299] Negotiation successful
0812 09:49:27.512998 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.99.2@KRBTEST.COM" created.
I20260812 09:49:27.514048 14175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:27.500281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:46849 (local address 127.13.99.1:44775)
0812 09:49:27.501158 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:27.501217 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0812 09:49:27.502419 (+ 1202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:49:27.503047 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:49:27.503054 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:49:27.503394 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:49:27.503855 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:49:27.503867 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:49:27.504816 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:49:27.504819 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:49:27.505700 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:49:27.505709 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:49:27.506240 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:49:27.506244 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:49:27.506479 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:27.507157 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:49:27.507183 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:49:27.509016 (+ 1833us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:49:27.511119 (+ 2103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:49:27.511124 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:49:27.511135 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:49:27.511393 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:49:27.511745 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:49:27.511749 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:49:27.511751 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:49:27.511889 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:49:27.512315 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:49:27.512322 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:49:27.512633 (+ 311us) client_negotiation.cc:769] Sending connection context
0812 09:49:27.512822 (+ 189us) client_negotiation.cc:240] Negotiation successful
0812 09:49:27.513140 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":168,"threads_started":1}
I20260812 09:49:27.515285 14173 heartbeater.cc:347] Connected to a master server at 127.13.99.62:46849
I20260812 09:49:27.515542 14173 heartbeater.cc:464] Registering TS with master...
I20260812 09:49:27.516031 14173 heartbeater.cc:511] Master 127.13.99.62:46849 requested a full tablet report, sending...
I20260812 09:49:27.517584 13862 ts_manager.cc:194] Registered new tserver with Master: ce0c794ffd2c4812b9d591ffe17aeb6f (127.13.99.1:36301)
I20260812 09:49:27.518697 13862 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.99.1@KRBTEST.COM'} at 127.13.99.1:44775
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.99.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.2.keytab.
Entry for principal kudu/127.13.99.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.2.keytab.
I20260812 09:49:27.531143 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.99.2:0
--local_ip_for_outbound_sockets=127.13.99.2
--webserver_interface=127.13.99.2
--webserver_port=0
--tserver_master_addrs=127.13.99.62:46849
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.2.keytab
--principal=kudu/127.13.99.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:45893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:49:27.646821 14180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:49:27.647145 14180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:49:27.647192 14180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:49:27.651117 14180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:49:27.651166 14180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:49:27.651221 14180 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.99.2
I20260812 09:49:27.656248 14180 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:45893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.2.keytab
--principal=kudu/127.13.99.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.99.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.99.2
--webserver_port=0
--tserver_master_addrs=127.13.99.62:46849
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14180
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.99.2
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:49:27.657277 14180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:49:27.658237 14180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:49:27.667740 14195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:49:27.667814 14180 server_base.cc:1034] running on GCE node
W20260812 09:49:27.667806 14192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:49:27.667762 14193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:49:27.668380 14180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:49:27.669116 14180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:49:27.670274 14180 hybrid_clock.cc:648] HybridClock initialized: now 1786528167670257 us; error 36 us; skew 500 ppm
Aug 12 09:49:27 dist-test-slave-jq4t krb5kdc[13788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528167, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:49:27.673125 14180 init.cc:377] Logged in from keytab as kudu/127.13.99.2@KRBTEST.COM (short username kudu)
I20260812 09:49:27.673959 14180 webserver.cc:460] Webserver started at http://127.13.99.2:33351/ using document root <none> and password file <none>
I20260812 09:49:27.674464 14180 fs_manager.cc:359] Metadata directory not provided
I20260812 09:49:27.674525 14180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:49:27.674685 14180 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:49:27.676657 14180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/minicluster-data/ts-1/data/instance:
uuid: "e9dc649f2202402dba21757c8ec07eec"
format_stamp: "Formatted at 2026-08-12 09:49:27 on dist-test-slave-jq4t"
server_key: "cb83b2edfeaf43465e2a094d7b4dd6f8"
server_key_iv: "27338bb87945a0310848807120862eb2"
server_key_version: "encryptionkey@0"
I20260812 09:49:27.677156 14180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/minicluster-data/ts-1/wal/instance:
uuid: "e9dc649f2202402dba21757c8ec07eec"
format_stamp: "Formatted at 2026-08-12 09:49:27 on dist-test-slave-jq4t"
server_key: "cb83b2edfeaf43465e2a094d7b4dd6f8"
server_key_iv: "27338bb87945a0310848807120862eb2"
server_key_version: "encryptionkey@0"
I20260812 09:49:27.680917 14180 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260812 09:49:27.683179 14209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:49:27.684363 14180 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 09:49:27.684517 14180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/minicluster-data/ts-1/wal
uuid: "e9dc649f2202402dba21757c8ec07eec"
format_stamp: "Formatted at 2026-08-12 09:49:27 on dist-test-slave-jq4t"
server_key: "cb83b2edfeaf43465e2a094d7b4dd6f8"
server_key_iv: "27338bb87945a0310848807120862eb2"
server_key_version: "encryptionkey@0"
I20260812 09:49:27.684634 14180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:49:27.696158 14180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:49:27.699076 14180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:49:27.699267 14180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:49:27.700014 14180 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 09:49:27.700965 14180 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:49:27.701043 14180 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 09:49:27.701112 14180 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:49:27.701164 14180 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 09:49:27.709980 14180 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.2:39957
I20260812 09:49:27.710013 14391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.2:39957 every 8 connection(s)
I20260812 09:49:27.710891 14180 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/minicluster-data/ts-1/data/info.pb
I20260812 09:49:27.718233 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 14180
I20260812 09:49:27.718406 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/minicluster-data/ts-1/wal/instance
I20260812 09:49:27.718729 13708 external_mini_cluster.cc:1428] Setting key e1a998c7d485696c740023675167fcd2
Aug 12 09:49:27 dist-test-slave-jq4t krb5kdc[13788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528167, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.2@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
I20260812 09:49:27.726532 13937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:27.712910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.2:32793 (local address 127.13.99.62:46849)
0812 09:49:27.713063 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:27.713067 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:27.714233 (+ 1166us) server_negotiation.cc:407] Connection header received
0812 09:49:27.715192 (+ 959us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:27.715196 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:27.715261 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:27.715368 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:49:27.716685 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:27.717235 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:27.718505 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:27.718744 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:27.718868 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:27.722181 (+ 3313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:49:27.722185 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:49:27.722186 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:49:27.722212 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:49:27.724101 (+ 1889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:49:27.724755 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:49:27.724762 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:49:27.724766 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:49:27.724873 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:49:27.725239 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:49:27.725245 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:49:27.725250 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:49:27.725521 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:49:27.725630 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:27.726165 (+ 535us) server_negotiation.cc:299] Negotiation successful
0812 09:49:27.726317 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260812 09:49:27.727141 14396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:27.713339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:46849 (local address 127.13.99.2:32793)
0812 09:49:27.714062 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:27.714105 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0812 09:49:27.714968 (+ 863us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:49:27.715577 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:49:27.715585 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:49:27.715960 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:49:27.716527 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:49:27.716553 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:49:27.717379 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:49:27.717385 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:49:27.718307 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:49:27.718316 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:49:27.718919 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:49:27.718923 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:49:27.719095 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:27.720064 (+ 969us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:49:27.720088 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:49:27.722003 (+ 1915us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:49:27.724239 (+ 2236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:49:27.724245 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:49:27.724258 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:49:27.724648 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:49:27.724971 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:49:27.724973 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:49:27.724974 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:49:27.725134 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:49:27.725626 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:49:27.725631 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:49:27.725882 (+ 251us) client_negotiation.cc:769] Sending connection context
0812 09:49:27.726074 (+ 192us) client_negotiation.cc:240] Negotiation successful
0812 09:49:27.726384 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":130,"threads_started":1}
I20260812 09:49:27.728328 14392 heartbeater.cc:347] Connected to a master server at 127.13.99.62:46849
I20260812 09:49:27.728616 14392 heartbeater.cc:464] Registering TS with master...
I20260812 09:49:27.729133 14392 heartbeater.cc:511] Master 127.13.99.62:46849 requested a full tablet report, sending...
I20260812 09:49:27.730279 13862 ts_manager.cc:194] Registered new tserver with Master: e9dc649f2202402dba21757c8ec07eec (127.13.99.2:39957)
WARNING: no policy specified for kudu/127.13.99.3@KRBTEST.COM; defaulting to no policy
I20260812 09:49:27.731057 13862 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.99.2@KRBTEST.COM'} at 127.13.99.2:32793
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.99.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.99.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.3.keytab.
Entry for principal kudu/127.13.99.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.3.keytab.
I20260812 09:49:27.754247 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.99.3:0
--local_ip_for_outbound_sockets=127.13.99.3
--webserver_interface=127.13.99.3
--webserver_port=0
--tserver_master_addrs=127.13.99.62:46849
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.3.keytab
--principal=kudu/127.13.99.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:45893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:49:27.873975 14402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:49:27.874282 14402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:49:27.874378 14402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:49:27.878971 14402 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:49:27.879055 14402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:49:27.879149 14402 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.99.3
I20260812 09:49:27.885079 14402 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:45893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.3.keytab
--principal=kudu/127.13.99.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.99.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.99.3
--webserver_port=0
--tserver_master_addrs=127.13.99.62:46849
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14402
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.99.3
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:49:27.886183 14402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:49:27.887280 14402 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:49:27.897419 14409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:49:27.897437 14408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:49:27.897629 14411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:49:27.897804 14402 server_base.cc:1034] running on GCE node
I20260812 09:49:27.898233 14402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:49:27.898854 14402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:49:27.900015 14402 hybrid_clock.cc:648] HybridClock initialized: now 1786528167899998 us; error 34 us; skew 500 ppm
Aug 12 09:49:27 dist-test-slave-jq4t krb5kdc[13788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528167, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:49:27.903041 14402 init.cc:377] Logged in from keytab as kudu/127.13.99.3@KRBTEST.COM (short username kudu)
I20260812 09:49:27.903929 14402 webserver.cc:460] Webserver started at http://127.13.99.3:36111/ using document root <none> and password file <none>
I20260812 09:49:27.904572 14402 fs_manager.cc:359] Metadata directory not provided
I20260812 09:49:27.904666 14402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:49:27.904910 14402 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:49:27.907055 14402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/minicluster-data/ts-2/data/instance:
uuid: "0574b5fade354afc832db5f65c5ed6ad"
format_stamp: "Formatted at 2026-08-12 09:49:27 on dist-test-slave-jq4t"
server_key: "d34ff5808384193d7856dee9b1782541"
server_key_iv: "153c94c8d9574caa23f3d36fac74502a"
server_key_version: "encryptionkey@0"
I20260812 09:49:27.907613 14402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/minicluster-data/ts-2/wal/instance:
uuid: "0574b5fade354afc832db5f65c5ed6ad"
format_stamp: "Formatted at 2026-08-12 09:49:27 on dist-test-slave-jq4t"
server_key: "d34ff5808384193d7856dee9b1782541"
server_key_iv: "153c94c8d9574caa23f3d36fac74502a"
server_key_version: "encryptionkey@0"
I20260812 09:49:27.911558 14402 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260812 09:49:27.914100 14421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:49:27.915369 14402 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260812 09:49:27.915508 14402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/minicluster-data/ts-2/wal
uuid: "0574b5fade354afc832db5f65c5ed6ad"
format_stamp: "Formatted at 2026-08-12 09:49:27 on dist-test-slave-jq4t"
server_key: "d34ff5808384193d7856dee9b1782541"
server_key_iv: "153c94c8d9574caa23f3d36fac74502a"
server_key_version: "encryptionkey@0"
I20260812 09:49:27.915622 14402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:49:27.951813 14402 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:49:27.955071 14402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:49:27.955279 14402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:49:27.955897 14402 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 09:49:27.956807 14402 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:49:27.956877 14402 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 09:49:27.956950 14402 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:49:27.957002 14402 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 09:49:27.966699 14402 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.3:40267
I20260812 09:49:27.966728 14619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.3:40267 every 8 connection(s)
I20260812 09:49:27.967577 14402 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/minicluster-data/ts-2/data/info.pb
I20260812 09:49:27.972384 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 14402
I20260812 09:49:27.972560 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/minicluster-data/ts-2/wal/instance
I20260812 09:49:27.972868 13708 external_mini_cluster.cc:1428] Setting key f965dfaaa9ae3317527cf4c39b520f6b
Aug 12 09:49:27 dist-test-slave-jq4t krb5kdc[13788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528167, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.3@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
I20260812 09:49:27.981606 13937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:27.969362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.3:54937 (local address 127.13.99.62:46849)
0812 09:49:27.969518 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:27.969522 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:27.970579 (+ 1057us) server_negotiation.cc:407] Connection header received
0812 09:49:27.971970 (+ 1391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:27.971973 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:27.972033 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:27.972168 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:49:27.973533 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:27.974080 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:27.975252 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:27.975484 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:27.975602 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:27.978015 (+ 2413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:49:27.978019 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:49:27.978022 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:49:27.978055 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:49:27.979405 (+ 1350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:49:27.980010 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:49:27.980015 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:49:27.980017 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:49:27.980104 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:49:27.980455 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:49:27.980459 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:49:27.980461 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:49:27.980689 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:49:27.980807 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:27.981299 (+ 492us) server_negotiation.cc:299] Negotiation successful
0812 09:49:27.981436 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260812 09:49:27.982362 14624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:27.969795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:46849 (local address 127.13.99.3:54937)
0812 09:49:27.970422 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:27.970481 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0812 09:49:27.971748 (+ 1267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:49:27.972333 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:49:27.972341 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:49:27.972827 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:49:27.973372 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:49:27.973387 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:49:27.974215 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:49:27.974219 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:49:27.975114 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:49:27.975124 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:49:27.975641 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:49:27.975645 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:49:27.975821 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:27.976429 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:49:27.976445 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:49:27.977841 (+ 1396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:49:27.979530 (+ 1689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:49:27.979536 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:49:27.979551 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:49:27.979886 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:49:27.980203 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:49:27.980207 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:49:27.980209 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:49:27.980351 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:49:27.980805 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:49:27.980812 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:49:27.981066 (+ 254us) client_negotiation.cc:769] Sending connection context
0812 09:49:27.981256 (+ 190us) client_negotiation.cc:240] Negotiation successful
0812 09:49:27.981571 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":109,"threads_started":1}
I20260812 09:49:27.983464 14620 heartbeater.cc:347] Connected to a master server at 127.13.99.62:46849
I20260812 09:49:27.983680 14620 heartbeater.cc:464] Registering TS with master...
I20260812 09:49:27.984153 14620 heartbeater.cc:511] Master 127.13.99.62:46849 requested a full tablet report, sending...
I20260812 09:49:27.985296 13862 ts_manager.cc:194] Registered new tserver with Master: 0574b5fade354afc832db5f65c5ed6ad (127.13.99.3:40267)
I20260812 09:49:27.985984 13862 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.99.3@KRBTEST.COM'} at 127.13.99.3:54937
I20260812 09:49:27.986598 13708 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 09:49:27.996649 13937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:27.988969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51428 (local address 127.13.99.62:46849)
0812 09:49:27.989111 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:27.989116 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:27.989312 (+ 196us) server_negotiation.cc:407] Connection header received
0812 09:49:27.989460 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:27.989463 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:27.989510 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:27.989594 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:49:27.990303 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:27.990801 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:27.991689 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:27.991908 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:27.992017 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:27.993465 (+ 1448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:49:27.993469 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:49:27.993472 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:49:27.993503 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:49:27.994856 (+ 1353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:49:27.995296 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:49:27.995300 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:49:27.995302 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:49:27.995394 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:49:27.995691 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:49:27.995694 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:49:27.995696 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:49:27.995919 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:49:27.996066 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:27.996317 (+ 251us) server_negotiation.cc:299] Negotiation successful
0812 09:49:27.996446 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260812 09:49:27.999598 13861 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51428:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 09:49:28.002660 13861 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 09:49:28.017002 14645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:28.011113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58206 (local address 127.13.99.3:40267)
0812 09:49:28.012120 (+ 1007us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:28.012136 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:28.012164 (+ 28us) server_negotiation.cc:407] Connection header received
0812 09:49:28.012245 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:28.012251 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:28.012506 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:28.012700 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:49:28.012993 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:28.013944 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:28.015444 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:28.016267 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:28.016390 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:28.016567 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:28.016746 (+ 179us) server_negotiation.cc:299] Negotiation successful
0812 09:49:28.016847 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":896,"thread_start_us":164,"threads_started":1}
I20260812 09:49:28.016990 14641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:28.010983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.3:40267 (local address 127.0.0.1:58206)
0812 09:49:28.011463 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:28.011508 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 09:49:28.011650 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:49:28.012694 (+ 1044us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:49:28.012700 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:49:28.012720 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:49:28.012866 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:49:28.012878 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:49:28.014075 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:49:28.014078 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:49:28.015289 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:49:28.015301 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:49:28.016391 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:49:28.016395 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:49:28.016532 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:28.016578 (+ 46us) client_negotiation.cc:769] Sending connection context
0812 09:49:28.016698 (+ 120us) client_negotiation.cc:240] Negotiation successful
0812 09:49:28.016836 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":83,"threads_started":1}
I20260812 09:49:28.017331 14646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:28.011401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.2:39957 (local address 127.0.0.1:57056)
0812 09:49:28.011667 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:28.011684 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 09:49:28.011780 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:49:28.012827 (+ 1047us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:49:28.012831 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:49:28.012849 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:49:28.012975 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:49:28.012982 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:49:28.014597 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:49:28.014601 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:49:28.015828 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:49:28.015839 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:49:28.016895 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:49:28.016899 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:49:28.017019 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:28.017052 (+ 33us) client_negotiation.cc:769] Sending connection context
0812 09:49:28.017129 (+ 77us) client_negotiation.cc:240] Negotiation successful
0812 09:49:28.017197 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":48,"thread_start_us":121,"threads_started":1}
I20260812 09:49:28.018041 14648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:28.011691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.1:36301 (local address 127.0.0.1:42538)
0812 09:49:28.012421 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:28.012434 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 09:49:28.012554 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:49:28.013371 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:49:28.013374 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:49:28.013398 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:49:28.013492 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:49:28.013497 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:49:28.014977 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:49:28.014980 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:49:28.016035 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:49:28.016046 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:49:28.017619 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:49:28.017623 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:49:28.017724 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:28.017747 (+ 23us) client_negotiation.cc:769] Sending connection context
0812 09:49:28.017819 (+ 72us) client_negotiation.cc:240] Negotiation successful
0812 09:49:28.017893 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":671,"mutex_wait_us":32,"thread_start_us":129,"threads_started":1}
I20260812 09:49:28.018054 14649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:28.011873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57056 (local address 127.13.99.2:39957)
0812 09:49:28.012306 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:28.012318 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:28.012342 (+ 24us) server_negotiation.cc:407] Connection header received
0812 09:49:28.012417 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:28.012423 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:28.012658 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:28.013128 (+ 470us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:49:28.013292 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:28.014442 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:28.015959 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:28.016766 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:28.017604 (+ 838us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:28.017740 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:28.017797 (+ 57us) server_negotiation.cc:299] Negotiation successful
0812 09:49:28.017879 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":136,"threads_started":1}
I20260812 09:49:28.018497 14650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:28.012048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42538 (local address 127.13.99.1:36301)
0812 09:49:28.012875 (+ 827us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:28.012891 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:28.012914 (+ 23us) server_negotiation.cc:407] Connection header received
0812 09:49:28.012989 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:28.012994 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:28.013206 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:28.013359 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:49:28.013595 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:28.014680 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:28.016166 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:28.017031 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:28.017406 (+ 375us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:28.017567 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:28.018253 (+ 686us) server_negotiation.cc:299] Negotiation successful
0812 09:49:28.018351 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":724,"thread_start_us":65,"threads_started":1}
I20260812 09:49:28.020390 14493 tablet_service.cc:1467] Processing CreateTablet for tablet 13a292a4fd6e4a15a483bdfa5a09fb78 (DEFAULT_TABLE table=test-table [id=ea6aa5c47cb44f86a2a63d08b88280d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 09:49:28.021624 14493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13a292a4fd6e4a15a483bdfa5a09fb78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:49:28.021260 14070 tablet_service.cc:1467] Processing CreateTablet for tablet 13a292a4fd6e4a15a483bdfa5a09fb78 (DEFAULT_TABLE table=test-table [id=ea6aa5c47cb44f86a2a63d08b88280d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 09:49:28.021540 14285 tablet_service.cc:1467] Processing CreateTablet for tablet 13a292a4fd6e4a15a483bdfa5a09fb78 (DEFAULT_TABLE table=test-table [id=ea6aa5c47cb44f86a2a63d08b88280d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 09:49:28.022238 14070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13a292a4fd6e4a15a483bdfa5a09fb78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:49:28.022588 14285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13a292a4fd6e4a15a483bdfa5a09fb78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:49:28.028410 14652 tablet_bootstrap.cc:492] T 13a292a4fd6e4a15a483bdfa5a09fb78 P e9dc649f2202402dba21757c8ec07eec: Bootstrap starting.
I20260812 09:49:28.029053 14654 tablet_bootstrap.cc:492] T 13a292a4fd6e4a15a483bdfa5a09fb78 P ce0c794ffd2c4812b9d591ffe17aeb6f: Bootstrap starting.
I20260812 09:49:28.029364 14656 tablet_bootstrap.cc:492] T 13a292a4fd6e4a15a483bdfa5a09fb78 P 0574b5fade354afc832db5f65c5ed6ad: Bootstrap starting.
I20260812 09:49:28.031364 14656 tablet_bootstrap.cc:654] T 13a292a4fd6e4a15a483bdfa5a09fb78 P 0574b5fade354afc832db5f65c5ed6ad: Neither blocks nor log segments found. Creating new log.
I20260812 09:49:28.031384 14654 tablet_bootstrap.cc:654] T 13a292a4fd6e4a15a483bdfa5a09fb78 P ce0c794ffd2c4812b9d591ffe17aeb6f: Neither blocks nor log segments found. Creating new log.
I20260812 09:49:28.031383 14652 tablet_bootstrap.cc:654] T 13a292a4fd6e4a15a483bdfa5a09fb78 P e9dc649f2202402dba21757c8ec07eec: Neither blocks nor log segments found. Creating new log.
I20260812 09:49:28.032338 14654 log.cc:824] T 13a292a4fd6e4a15a483bdfa5a09fb78 P ce0c794ffd2c4812b9d591ffe17aeb6f: Log is configured to *not* fsync() on all Append() calls
I20260812 09:49:28.032511 14656 log.cc:824] T 13a292a4fd6e4a15a483bdfa5a09fb78 P 0574b5fade354afc832db5f65c5ed6ad: Log is configured to *not* fsync() on all Append() calls
I20260812 09:49:28.033174 14652 log.cc:824] T 13a292a4fd6e4a15a483bdfa5a09fb78 P e9dc649f2202402dba21757c8ec07eec: Log is configured to *not* fsync() on all Append() calls
I20260812 09:49:28.034389 14654 tablet_bootstrap.cc:492] T 13a292a4fd6e4a15a483bdfa5a09fb78 P ce0c794ffd2c4812b9d591ffe17aeb6f: No bootstrap required, opened a new log
I20260812 09:49:28.034603 14654 ts_tablet_manager.cc:1397] T 13a292a4fd6e4a15a483bdfa5a09fb78 P ce0c794ffd2c4812b9d591ffe17aeb6f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260812 09:49:28.035028 14652 tablet_bootstrap.cc:492] T 13a292a4fd6e4a15a483bdfa5a09fb78 P e9dc649f2202402dba21757c8ec07eec: No bootstrap required, opened a new log
I20260812 09:49:28.035025 14656 tablet_bootstrap.cc:492] T 13a292a4fd6e4a15a483bdfa5a09fb78 P 0574b5fade354afc832db5f65c5ed6ad: No bootstrap required, opened a new log
I20260812 09:49:28.035209 14656 ts_tablet_manager.cc:1397] T 13a292a4fd6e4a15a483bdfa5a09fb78 P 0574b5fade354afc832db5f65c5ed6ad: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260812 09:49:28.035209 14652 ts_tablet_manager.cc:1397] T 13a292a4fd6e4a15a483bdfa5a09fb78 P e9dc649f2202402dba21757c8ec07eec: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20260812 09:49:28.038079 14654 raft_consensus.cc:348] T 13a292a4fd6e4a15a483bdfa5a09fb78 P ce0c794ffd2c4812b9d591ffe17aeb6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0574b5fade354afc832db5f65c5ed6ad" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 40267 } } peers { permanent_uuid: "ce0c794ffd2c4812b9d591ffe17aeb6f" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 36301 } } peers { permanent_uuid: "e9dc649f2202402dba21757c8ec07eec" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 39957 } }
I20260812 09:49:28.038326 14654 raft_consensus.cc:374] T 13a292a4fd6e4a15a483bdfa5a09fb78 P ce0c794ffd2c4812b9d591ffe17aeb6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:49:28.038384 14654 raft_consensus.cc:729] T 13a292a4fd6e4a15a483bdfa5a09fb78 P ce0c794ffd2c4812b9d591ffe17aeb6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce0c794ffd2c4812b9d591ffe17aeb6f, State: Initialized, Role: FOLLOWER
I20260812 09:49:28.038863 14654 consensus_queue.cc:260] T 13a292a4fd6e4a15a483bdfa5a09fb78 P ce0c794ffd2c4812b9d591ffe17aeb6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0574b5fade354afc832db5f65c5ed6ad" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 40267 } } peers { permanent_uuid: "ce0c794ffd2c4812b9d591ffe17aeb6f" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 36301 } } peers { permanent_uuid: "e9dc649f2202402dba21757c8ec07eec" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 39957 } }
I20260812 09:49:28.038966 14656 raft_consensus.cc:348] T 13a292a4fd6e4a15a483bdfa5a09fb78 P 0574b5fade354afc832db5f65c5ed6ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0574b5fade354afc832db5f65c5ed6ad" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 40267 } } peers { permanent_uuid: "ce0c794ffd2c4812b9d591ffe17aeb6f" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 36301 } } peers { permanent_uuid: "e9dc649f2202402dba21757c8ec07eec" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 39957 } }
I20260812 09:49:28.039227 14656 raft_consensus.cc:374] T 13a292a4fd6e4a15a483bdfa5a09fb78 P 0574b5fade354afc832db5f65c5ed6ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:49:28.039280 14656 raft_consensus.cc:729] T 13a292a4fd6e4a15a483bdfa5a09fb78 P 0574b5fade354afc832db5f65c5ed6ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0574b5fade354afc832db5f65c5ed6ad, State: Initialized, Role: FOLLOWER
I20260812 09:49:28.038966 14652 raft_consensus.cc:348] T 13a292a4fd6e4a15a483bdfa5a09fb78 P e9dc649f2202402dba21757c8ec07eec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0574b5fade354afc832db5f65c5ed6ad" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 40267 } } peers { permanent_uuid: "ce0c794ffd2c4812b9d591ffe17aeb6f" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 36301 } } peers { permanent_uuid: "e9dc649f2202402dba21757c8ec07eec" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 39957 } }
I20260812 09:49:28.039373 14652 raft_consensus.cc:374] T 13a292a4fd6e4a15a483bdfa5a09fb78 P e9dc649f2202402dba21757c8ec07eec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:49:28.039417 14652 raft_consensus.cc:729] T 13a292a4fd6e4a15a483bdfa5a09fb78 P e9dc649f2202402dba21757c8ec07eec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9dc649f2202402dba21757c8ec07eec, State: Initialized, Role: FOLLOWER
I20260812 09:49:28.039688 14656 consensus_queue.cc:260] T 13a292a4fd6e4a15a483bdfa5a09fb78 P 0574b5fade354afc832db5f65c5ed6ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0574b5fade354afc832db5f65c5ed6ad" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 40267 } } peers { permanent_uuid: "ce0c794ffd2c4812b9d591ffe17aeb6f" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 36301 } } peers { permanent_uuid: "e9dc649f2202402dba21757c8ec07eec" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 39957 } }
I20260812 09:49:28.039739 14652 consensus_queue.cc:260] T 13a292a4fd6e4a15a483bdfa5a09fb78 P e9dc649f2202402dba21757c8ec07eec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0574b5fade354afc832db5f65c5ed6ad" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 40267 } } peers { permanent_uuid: "ce0c794ffd2c4812b9d591ffe17aeb6f" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 36301 } } peers { permanent_uuid: "e9dc649f2202402dba21757c8ec07eec" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 39957 } }
I20260812 09:49:28.039934 14654 ts_tablet_manager.cc:1428] T 13a292a4fd6e4a15a483bdfa5a09fb78 P ce0c794ffd2c4812b9d591ffe17aeb6f: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260812 09:49:28.040339 14173 heartbeater.cc:503] Master 127.13.99.62:46849 was elected leader, sending a full tablet report...
I20260812 09:49:28.040634 14656 ts_tablet_manager.cc:1428] T 13a292a4fd6e4a15a483bdfa5a09fb78 P 0574b5fade354afc832db5f65c5ed6ad: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260812 09:49:28.040634 14652 ts_tablet_manager.cc:1428] T 13a292a4fd6e4a15a483bdfa5a09fb78 P e9dc649f2202402dba21757c8ec07eec: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260812 09:49:28.040941 14620 heartbeater.cc:503] Master 127.13.99.62:46849 was elected leader, sending a full tablet report...
I20260812 09:49:28.041430 14392 heartbeater.cc:503] Master 127.13.99.62:46849 was elected leader, sending a full tablet report...
W20260812 09:49:28.212100 14393 tablet.cc:2369] T 13a292a4fd6e4a15a483bdfa5a09fb78 P e9dc649f2202402dba21757c8ec07eec: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 09:49:28.218482 14621 tablet.cc:2369] T 13a292a4fd6e4a15a483bdfa5a09fb78 P 0574b5fade354afc832db5f65c5ed6ad: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 09:49:28.249418 14174 tablet.cc:2369] T 13a292a4fd6e4a15a483bdfa5a09fb78 P ce0c794ffd2c4812b9d591ffe17aeb6f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 09:49:28.310670 14669 raft_consensus.cc:482] T 13a292a4fd6e4a15a483bdfa5a09fb78 P 0574b5fade354afc832db5f65c5ed6ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 09:49:28.310886 14669 raft_consensus.cc:504] T 13a292a4fd6e4a15a483bdfa5a09fb78 P 0574b5fade354afc832db5f65c5ed6ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0574b5fade354afc832db5f65c5ed6ad" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 40267 } } peers { permanent_uuid: "ce0c794ffd2c4812b9d591ffe17aeb6f" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 36301 } } peers { permanent_uuid: "e9dc649f2202402dba21757c8ec07eec" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 39957 } }
I20260812 09:49:28.312011 14669 leader_election.cc:288] T 13a292a4fd6e4a15a483bdfa5a09fb78 P 0574b5fade354afc832db5f65c5ed6ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce0c794ffd2c4812b9d591ffe17aeb6f (127.13.99.1:36301), e9dc649f2202402dba21757c8ec07eec (127.13.99.2:39957)
I20260812 09:49:28.317467 14649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:28.312545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.3:36257 (local address 127.13.99.2:39957)
0812 09:49:28.312682 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:28.312689 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:28.312869 (+ 180us) server_negotiation.cc:407] Connection header received
0812 09:49:28.312957 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:28.312960 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:28.313009 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:28.313085 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:49:28.313346 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:28.314114 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:28.316053 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:28.316712 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:28.316829 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:28.316897 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:28.317244 (+ 347us) server_negotiation.cc:299] Negotiation successful
0812 09:49:28.317331 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260812 09:49:28.317467 14650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:28.312409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.3:34953 (local address 127.13.99.1:36301)
0812 09:49:28.312565 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:28.312572 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:28.312589 (+ 17us) server_negotiation.cc:407] Connection header received
0812 09:49:28.312795 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:28.312798 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:28.312875 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:28.312978 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:49:28.313309 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:28.314118 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:28.316053 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:28.316719 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:28.316828 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:28.316897 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:28.317244 (+ 347us) server_negotiation.cc:299] Negotiation successful
0812 09:49:28.317332 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 09:49:28.317477 14624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:28.312268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.1:36301 (local address 127.13.99.3:34953)
0812 09:49:28.312467 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:28.312537 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0812 09:49:28.312649 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:49:28.312997 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:49:28.313000 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:49:28.313015 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:49:28.313170 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:49:28.313177 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:49:28.314243 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:49:28.314246 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:49:28.315918 (+ 1672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:49:28.315927 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:49:28.316860 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:49:28.316864 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:49:28.317026 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:28.317086 (+ 60us) client_negotiation.cc:769] Sending connection context
0812 09:49:28.317183 (+ 97us) client_negotiation.cc:240] Negotiation successful
0812 09:49:28.317303 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74,"mutex_wait_us":343}
I20260812 09:49:28.317487 14674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:28.312268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.2:39957 (local address 127.13.99.3:36257)
0812 09:49:28.312773 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:28.312788 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 09:49:28.312871 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:49:28.313105 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:49:28.313107 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:49:28.313120 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:49:28.313230 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:49:28.313239 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:49:28.314243 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:49:28.314246 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:49:28.315889 (+ 1643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:49:28.315927 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:49:28.316860 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:49:28.316864 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:49:28.317038 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:28.317079 (+ 41us) client_negotiation.cc:769] Sending connection context
0812 09:49:28.317183 (+ 104us) client_negotiation.cc:240] Negotiation successful
0812 09:49:28.317302 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"mutex_wait_us":301,"thread_start_us":144,"threads_started":1}
I20260812 09:49:28.318336 14103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13a292a4fd6e4a15a483bdfa5a09fb78" candidate_uuid: "0574b5fade354afc832db5f65c5ed6ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce0c794ffd2c4812b9d591ffe17aeb6f" is_pre_election: true
I20260812 09:49:28.318334 14320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13a292a4fd6e4a15a483bdfa5a09fb78" candidate_uuid: "0574b5fade354afc832db5f65c5ed6ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9dc649f2202402dba21757c8ec07eec" is_pre_election: true
I20260812 09:49:28.318686 14320 raft_consensus.cc:2454] T 13a292a4fd6e4a15a483bdfa5a09fb78 P e9dc649f2202402dba21757c8ec07eec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0574b5fade354afc832db5f65c5ed6ad in term 0.
I20260812 09:49:28.318686 14103 raft_consensus.cc:2454] T 13a292a4fd6e4a15a483bdfa5a09fb78 P ce0c794ffd2c4812b9d591ffe17aeb6f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0574b5fade354afc832db5f65c5ed6ad in term 0.
I20260812 09:49:28.319286 14423 leader_election.cc:302] T 13a292a4fd6e4a15a483bdfa5a09fb78 P 0574b5fade354afc832db5f65c5ed6ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0574b5fade354afc832db5f65c5ed6ad, ce0c794ffd2c4812b9d591ffe17aeb6f, e9dc649f2202402dba21757c8ec07eec; no voters:
I20260812 09:49:28.319623 14669 raft_consensus.cc:2781] T 13a292a4fd6e4a15a483bdfa5a09fb78 P 0574b5fade354afc832db5f65c5ed6ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 09:49:28.319720 14669 raft_consensus.cc:482] T 13a292a4fd6e4a15a483bdfa5a09fb78 P 0574b5fade354afc832db5f65c5ed6ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 09:49:28.319765 14669 raft_consensus.cc:3037] T 13a292a4fd6e4a15a483bdfa5a09fb78 P 0574b5fade354afc832db5f65c5ed6ad [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:49:28.321012 14669 raft_consensus.cc:504] T 13a292a4fd6e4a15a483bdfa5a09fb78 P 0574b5fade354afc832db5f65c5ed6ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0574b5fade354afc832db5f65c5ed6ad" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 40267 } } peers { permanent_uuid: "ce0c794ffd2c4812b9d591ffe17aeb6f" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 36301 } } peers { permanent_uuid: "e9dc649f2202402dba21757c8ec07eec" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 39957 } }
I20260812 09:49:28.321400 14669 leader_election.cc:288] T 13a292a4fd6e4a15a483bdfa5a09fb78 P 0574b5fade354afc832db5f65c5ed6ad [CANDIDATE]: Term 1 election: Requested vote from peers ce0c794ffd2c4812b9d591ffe17aeb6f (127.13.99.1:36301), e9dc649f2202402dba21757c8ec07eec (127.13.99.2:39957)
I20260812 09:49:28.321805 14103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13a292a4fd6e4a15a483bdfa5a09fb78" candidate_uuid: "0574b5fade354afc832db5f65c5ed6ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce0c794ffd2c4812b9d591ffe17aeb6f"
I20260812 09:49:28.321853 14320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13a292a4fd6e4a15a483bdfa5a09fb78" candidate_uuid: "0574b5fade354afc832db5f65c5ed6ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9dc649f2202402dba21757c8ec07eec"
I20260812 09:49:28.321957 14103 raft_consensus.cc:3037] T 13a292a4fd6e4a15a483bdfa5a09fb78 P ce0c794ffd2c4812b9d591ffe17aeb6f [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:49:28.321978 14320 raft_consensus.cc:3037] T 13a292a4fd6e4a15a483bdfa5a09fb78 P e9dc649f2202402dba21757c8ec07eec [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:49:28.323014 14320 raft_consensus.cc:2454] T 13a292a4fd6e4a15a483bdfa5a09fb78 P e9dc649f2202402dba21757c8ec07eec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0574b5fade354afc832db5f65c5ed6ad in term 1.
I20260812 09:49:28.323092 14103 raft_consensus.cc:2454] T 13a292a4fd6e4a15a483bdfa5a09fb78 P ce0c794ffd2c4812b9d591ffe17aeb6f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0574b5fade354afc832db5f65c5ed6ad in term 1.
I20260812 09:49:28.323374 14426 leader_election.cc:302] T 13a292a4fd6e4a15a483bdfa5a09fb78 P 0574b5fade354afc832db5f65c5ed6ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0574b5fade354afc832db5f65c5ed6ad, e9dc649f2202402dba21757c8ec07eec; no voters:
I20260812 09:49:28.323606 14669 raft_consensus.cc:2781] T 13a292a4fd6e4a15a483bdfa5a09fb78 P 0574b5fade354afc832db5f65c5ed6ad [term 1 FOLLOWER]: Leader election won for term 1
I20260812 09:49:28.323856 14669 raft_consensus.cc:686] T 13a292a4fd6e4a15a483bdfa5a09fb78 P 0574b5fade354afc832db5f65c5ed6ad [term 1 LEADER]: Becoming Leader. State: Replica: 0574b5fade354afc832db5f65c5ed6ad, State: Running, Role: LEADER
I20260812 09:49:28.324210 14669 consensus_queue.cc:237] T 13a292a4fd6e4a15a483bdfa5a09fb78 P 0574b5fade354afc832db5f65c5ed6ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0574b5fade354afc832db5f65c5ed6ad" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 40267 } } peers { permanent_uuid: "ce0c794ffd2c4812b9d591ffe17aeb6f" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 36301 } } peers { permanent_uuid: "e9dc649f2202402dba21757c8ec07eec" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 39957 } }
I20260812 09:49:28.327432 13860 catalog_manager.cc:5495] T 13a292a4fd6e4a15a483bdfa5a09fb78 P 0574b5fade354afc832db5f65c5ed6ad reported cstate change: term changed from 0 to 1, leader changed from <none> to 0574b5fade354afc832db5f65c5ed6ad (127.13.99.3). New cstate: current_term: 1 leader_uuid: "0574b5fade354afc832db5f65c5ed6ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0574b5fade354afc832db5f65c5ed6ad" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 40267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce0c794ffd2c4812b9d591ffe17aeb6f" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 36301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9dc649f2202402dba21757c8ec07eec" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 39957 } health_report { overall_health: UNKNOWN } } }
I20260812 09:49:28.379923 14645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:28.375723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58208 (local address 127.13.99.3:40267)
0812 09:49:28.375886 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:28.375889 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:28.375904 (+ 15us) server_negotiation.cc:407] Connection header received
0812 09:49:28.376060 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:28.376064 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:28.376128 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:28.376222 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 09:49:28.376510 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:28.377047 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:28.377960 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:28.378237 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:28.378331 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:28.378678 (+ 347us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 09:49:28.379251 (+ 573us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 09:49:28.379412 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:28.379653 (+ 241us) server_negotiation.cc:299] Negotiation successful
0812 09:49:28.379737 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260812 09:49:28.386713 14103 raft_consensus.cc:1261] T 13a292a4fd6e4a15a483bdfa5a09fb78 P ce0c794ffd2c4812b9d591ffe17aeb6f [term 1 FOLLOWER]: Refusing update from remote peer 0574b5fade354afc832db5f65c5ed6ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 09:49:28.386713 14320 raft_consensus.cc:1261] T 13a292a4fd6e4a15a483bdfa5a09fb78 P e9dc649f2202402dba21757c8ec07eec [term 1 FOLLOWER]: Refusing update from remote peer 0574b5fade354afc832db5f65c5ed6ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 09:49:28.387552 14675 consensus_queue.cc:1035] T 13a292a4fd6e4a15a483bdfa5a09fb78 P 0574b5fade354afc832db5f65c5ed6ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce0c794ffd2c4812b9d591ffe17aeb6f" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 36301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 09:49:28.387781 14669 consensus_queue.cc:1035] T 13a292a4fd6e4a15a483bdfa5a09fb78 P 0574b5fade354afc832db5f65c5ed6ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9dc649f2202402dba21757c8ec07eec" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 39957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 09:49:28.395063 14684 mvcc.cc:205] Tried to move back new op lower bound from 7317619377702060032 to 7317619377458823168. Current Snapshot: MvccSnapshot[applied={T|T < 7317619377702060032}]
I20260812 09:49:28.396658 14689 mvcc.cc:205] Tried to move back new op lower bound from 7317619377702060032 to 7317619377458823168. Current Snapshot: MvccSnapshot[applied={T|T < 7317619377702060032}]
I20260812 09:49:28.404050 13862 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51428:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 09:49:28.404256 13862 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51428:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 09:49:28.406931 13862 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7fa7fc0aaebe44cfa7ee046a36462387: Sending DeleteTablet for 3 replicas of tablet 13a292a4fd6e4a15a483bdfa5a09fb78
I20260812 09:49:28.407850 14285 tablet_service.cc:1514] Processing DeleteTablet for tablet 13a292a4fd6e4a15a483bdfa5a09fb78 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 09:49:28 UTC) from {username='kudu', principal='kudu/127.13.99.62@KRBTEST.COM'} at 127.0.0.1:57056
I20260812 09:49:28.407833 14493 tablet_service.cc:1514] Processing DeleteTablet for tablet 13a292a4fd6e4a15a483bdfa5a09fb78 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 09:49:28 UTC) from {username='kudu', principal='kudu/127.13.99.62@KRBTEST.COM'} at 127.0.0.1:58206
I20260812 09:49:28.407879 14070 tablet_service.cc:1514] Processing DeleteTablet for tablet 13a292a4fd6e4a15a483bdfa5a09fb78 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 09:49:28 UTC) from {username='kudu', principal='kudu/127.13.99.62@KRBTEST.COM'} at 127.0.0.1:42538
I20260812 09:49:28.408566 14703 tablet_replica.cc:330] T 13a292a4fd6e4a15a483bdfa5a09fb78 P ce0c794ffd2c4812b9d591ffe17aeb6f: stopping tablet replica
I20260812 09:49:28.408915 14703 raft_consensus.cc:2229] T 13a292a4fd6e4a15a483bdfa5a09fb78 P ce0c794ffd2c4812b9d591ffe17aeb6f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 09:49:28.408947 14706 tablet_replica.cc:330] T 13a292a4fd6e4a15a483bdfa5a09fb78 P 0574b5fade354afc832db5f65c5ed6ad: stopping tablet replica
I20260812 09:49:28.409243 14706 raft_consensus.cc:2229] T 13a292a4fd6e4a15a483bdfa5a09fb78 P 0574b5fade354afc832db5f65c5ed6ad [term 1 LEADER]: Raft consensus shutting down.
I20260812 09:49:28.409269 14703 raft_consensus.cc:2258] T 13a292a4fd6e4a15a483bdfa5a09fb78 P ce0c794ffd2c4812b9d591ffe17aeb6f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 09:49:28.409394 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 13970
I20260812 09:49:28.409659 14706 raft_consensus.cc:2258] T 13a292a4fd6e4a15a483bdfa5a09fb78 P 0574b5fade354afc832db5f65c5ed6ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 09:49:28.410039 14707 tablet_replica.cc:330] T 13a292a4fd6e4a15a483bdfa5a09fb78 P e9dc649f2202402dba21757c8ec07eec: stopping tablet replica
I20260812 09:49:28.410395 14707 raft_consensus.cc:2229] T 13a292a4fd6e4a15a483bdfa5a09fb78 P e9dc649f2202402dba21757c8ec07eec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 09:49:28.410771 14707 raft_consensus.cc:2258] T 13a292a4fd6e4a15a483bdfa5a09fb78 P e9dc649f2202402dba21757c8ec07eec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 09:49:28.419904 14707 ts_tablet_manager.cc:1905] T 13a292a4fd6e4a15a483bdfa5a09fb78 P e9dc649f2202402dba21757c8ec07eec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 09:49:28.421780 14706 ts_tablet_manager.cc:1905] T 13a292a4fd6e4a15a483bdfa5a09fb78 P 0574b5fade354afc832db5f65c5ed6ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 09:49:28.423225 14707 ts_tablet_manager.cc:1918] T 13a292a4fd6e4a15a483bdfa5a09fb78 P e9dc649f2202402dba21757c8ec07eec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 09:49:28.423362 14707 log.cc:1196] T 13a292a4fd6e4a15a483bdfa5a09fb78 P e9dc649f2202402dba21757c8ec07eec: Deleting WAL directory at /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/minicluster-data/ts-1/wal/wals/13a292a4fd6e4a15a483bdfa5a09fb78
I20260812 09:49:28.423749 14707 ts_tablet_manager.cc:1939] T 13a292a4fd6e4a15a483bdfa5a09fb78 P e9dc649f2202402dba21757c8ec07eec: Deleting consensus metadata
I20260812 09:49:28.425066 14706 ts_tablet_manager.cc:1918] T 13a292a4fd6e4a15a483bdfa5a09fb78 P 0574b5fade354afc832db5f65c5ed6ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 09:49:28.425173 14706 log.cc:1196] T 13a292a4fd6e4a15a483bdfa5a09fb78 P 0574b5fade354afc832db5f65c5ed6ad: Deleting WAL directory at /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786527905902985-13708-0/minicluster-data/ts-2/wal/wals/13a292a4fd6e4a15a483bdfa5a09fb78
I20260812 09:49:28.425273 13839 catalog_manager.cc:4841] TS e9dc649f2202402dba21757c8ec07eec (127.13.99.2:39957): tablet 13a292a4fd6e4a15a483bdfa5a09fb78 (table test-table [id=ea6aa5c47cb44f86a2a63d08b88280d1]) successfully deleted
I20260812 09:49:28.425558 14706 ts_tablet_manager.cc:1939] T 13a292a4fd6e4a15a483bdfa5a09fb78 P 0574b5fade354afc832db5f65c5ed6ad: Deleting consensus metadata
I20260812 09:49:28.426625 13837 catalog_manager.cc:4841] TS 0574b5fade354afc832db5f65c5ed6ad (127.13.99.3:40267): tablet 13a292a4fd6e4a15a483bdfa5a09fb78 (table test-table [id=ea6aa5c47cb44f86a2a63d08b88280d1]) successfully deleted
W20260812 09:49:28.427204 13837 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.99.1:36301 (error 108)
I20260812 09:49:28.428181 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 14180
I20260812 09:49:28.428395 14648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:28.428006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.1:36301 (local address 127.0.0.1:42552)
0812 09:49:28.428168 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:28.428276 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.1:36301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260812 09:49:28.431006 13837 catalog_manager.cc:4584] TS ce0c794ffd2c4812b9d591ffe17aeb6f (127.13.99.1:36301): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 13a292a4fd6e4a15a483bdfa5a09fb78: Network error: Client connection negotiation failed: client connection to 127.13.99.1:36301: connect: Connection refused (error 111)
I20260812 09:49:28.443491 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 14402
I20260812 09:49:28.459956 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 13810
2026-08-12T09:49:28Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2982 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-12T09:49:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T09:49:28Z Disabled control of system clock
I20260812 09:49:28.506392 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786527905902985-13708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786527905902985-13708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786527905902985-13708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.62:43825
--webserver_interface=127.13.99.62
--webserver_port=0
--builtin_ntp_servers=127.13.99.20:32897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.99.62:43825
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 09:49:28.621050 14719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:49:28.621379 14719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:49:28.621454 14719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:49:28.625463 14719 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 09:49:28.625516 14719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:49:28.625531 14719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:49:28.625548 14719 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 09:49:28.625563 14719 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 09:49:28.630321 14719 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:32897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786527905902985-13708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786527905902985-13708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.99.62:43825
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.62:43825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.99.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786527905902985-13708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:49:28.631458 14719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:49:28.632908 14719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:49:28.641426 14727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:49:28.641459 14729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:49:28.641448 14731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:49:28.641819 14719 server_base.cc:1034] running on GCE node
I20260812 09:49:28.642457 14719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:49:28.643688 14719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:49:28.644863 14719 hybrid_clock.cc:648] HybridClock initialized: now 1786528168644841 us; error 35 us; skew 500 ppm
I20260812 09:49:28.650681 14719 webserver.cc:460] Webserver started at http://127.13.99.62:36441/ using document root <none> and password file <none>
I20260812 09:49:28.651326 14719 fs_manager.cc:359] Metadata directory not provided
I20260812 09:49:28.651393 14719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:49:28.651623 14719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:49:28.653824 14719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786527905902985-13708-0/minicluster-data/master-0/data/instance:
uuid: "88709457334645bcac18e28a01ca8c12"
format_stamp: "Formatted at 2026-08-12 09:49:28 on dist-test-slave-jq4t"
server_key: "98c7a6f8ad75d477d185d22d15e195b3"
server_key_iv: "9fb983e306d841990333641bab351f0b"
server_key_version: "encryptionkey@0"
I20260812 09:49:28.654412 14719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786527905902985-13708-0/minicluster-data/master-0/wal/instance:
uuid: "88709457334645bcac18e28a01ca8c12"
format_stamp: "Formatted at 2026-08-12 09:49:28 on dist-test-slave-jq4t"
server_key: "98c7a6f8ad75d477d185d22d15e195b3"
server_key_iv: "9fb983e306d841990333641bab351f0b"
server_key_version: "encryptionkey@0"
I20260812 09:49:28.658273 14719 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260812 09:49:28.660883 14737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:49:28.662191 14719 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260812 09:49:28.662325 14719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786527905902985-13708-0/minicluster-data/master-0/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786527905902985-13708-0/minicluster-data/master-0/wal
uuid: "88709457334645bcac18e28a01ca8c12"
format_stamp: "Formatted at 2026-08-12 09:49:28 on dist-test-slave-jq4t"
server_key: "98c7a6f8ad75d477d185d22d15e195b3"
server_key_iv: "9fb983e306d841990333641bab351f0b"
server_key_version: "encryptionkey@0"
I20260812 09:49:28.662461 14719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786527905902985-13708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786527905902985-13708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786527905902985-13708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:49:28.676520 14719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:49:28.677209 14719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:49:28.677418 14719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:49:28.684681 14719 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.62:43825
I20260812 09:49:28.684706 14828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.62:43825 every 8 connection(s)
I20260812 09:49:28.685647 14719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
I20260812 09:49:28.688951 14829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:49:28.692725 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 14719
I20260812 09:49:28.692858 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786527905902985-13708-0/minicluster-data/master-0/wal/instance
I20260812 09:49:28.693164 13708 external_mini_cluster.cc:1428] Setting key b2ed8cd2875ffe5dfbaff8073fcbbf99
I20260812 09:49:28.695027 14829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88709457334645bcac18e28a01ca8c12: Bootstrap starting.
I20260812 09:49:28.697571 14829 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88709457334645bcac18e28a01ca8c12: Neither blocks nor log segments found. Creating new log.
I20260812 09:49:28.698395 14829 log.cc:824] T 00000000000000000000000000000000 P 88709457334645bcac18e28a01ca8c12: Log is configured to *not* fsync() on all Append() calls
I20260812 09:49:28.700659 14829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88709457334645bcac18e28a01ca8c12: No bootstrap required, opened a new log
I20260812 09:49:28.702481 14836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:28.694650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39490 (local address 127.13.99.62:43825)
0812 09:49:28.695296 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:28.695313 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:28.695349 (+ 36us) server_negotiation.cc:407] Connection header received
0812 09:49:28.696008 (+ 659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:28.696024 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:28.696412 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:28.696757 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:49:28.697112 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:28.698155 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:28.699265 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:28.699721 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:28.699840 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:28.700756 (+ 916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:49:28.700764 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:49:28.700788 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 09:49:28.701184 (+ 396us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:49:28.701237 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 09:49:28.701258 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:49:28.701336 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:28.701514 (+ 178us) server_negotiation.cc:299] Negotiation successful
0812 09:49:28.701794 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":132,"threads_started":1}
I20260812 09:49:28.704651 14829 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88709457334645bcac18e28a01ca8c12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88709457334645bcac18e28a01ca8c12" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 43825 } }
I20260812 09:49:28.704949 14829 raft_consensus.cc:374] T 00000000000000000000000000000000 P 88709457334645bcac18e28a01ca8c12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:49:28.705018 14829 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88709457334645bcac18e28a01ca8c12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88709457334645bcac18e28a01ca8c12, State: Initialized, Role: FOLLOWER
I20260812 09:49:28.705482 14829 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88709457334645bcac18e28a01ca8c12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88709457334645bcac18e28a01ca8c12" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 43825 } }
I20260812 09:49:28.705633 14829 raft_consensus.cc:388] T 00000000000000000000000000000000 P 88709457334645bcac18e28a01ca8c12 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 09:49:28.705699 14829 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88709457334645bcac18e28a01ca8c12 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 09:49:28.705811 14829 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88709457334645bcac18e28a01ca8c12 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:49:28.706965 14829 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88709457334645bcac18e28a01ca8c12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88709457334645bcac18e28a01ca8c12" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 43825 } }
I20260812 09:49:28.707326 14829 leader_election.cc:302] T 00000000000000000000000000000000 P 88709457334645bcac18e28a01ca8c12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88709457334645bcac18e28a01ca8c12; no voters:
I20260812 09:49:28.707628 14829 leader_election.cc:288] T 00000000000000000000000000000000 P 88709457334645bcac18e28a01ca8c12 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 09:49:28.707804 14839 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 88709457334645bcac18e28a01ca8c12 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 09:49:28.708076 14839 raft_consensus.cc:686] T 00000000000000000000000000000000 P 88709457334645bcac18e28a01ca8c12 [term 1 LEADER]: Becoming Leader. State: Replica: 88709457334645bcac18e28a01ca8c12, State: Running, Role: LEADER
I20260812 09:49:28.708405 14839 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88709457334645bcac18e28a01ca8c12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88709457334645bcac18e28a01ca8c12" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 43825 } }
I20260812 09:49:28.708922 14829 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88709457334645bcac18e28a01ca8c12 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 09:49:28.710024 14845 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88709457334645bcac18e28a01ca8c12 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88709457334645bcac18e28a01ca8c12. Latest consensus state: current_term: 1 leader_uuid: "88709457334645bcac18e28a01ca8c12" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88709457334645bcac18e28a01ca8c12" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 43825 } } }
I20260812 09:49:28.710161 14845 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88709457334645bcac18e28a01ca8c12 [sys.catalog]: This master's current role is: LEADER
I20260812 09:49:28.710561 14858 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 09:49:28.711117 14841 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88709457334645bcac18e28a01ca8c12 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "88709457334645bcac18e28a01ca8c12" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88709457334645bcac18e28a01ca8c12" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 43825 } } }
I20260812 09:49:28.711248 14841 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88709457334645bcac18e28a01ca8c12 [sys.catalog]: This master's current role is: LEADER
I20260812 09:49:28.713948 14858 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 09:49:28.719449 14858 catalog_manager.cc:1347] Generated new cluster ID: 07a06d15b7b34a1d872d072c86c35d7e
I20260812 09:49:28.719522 14858 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 09:49:28.726689 14858 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 09:49:28.727609 14858 catalog_manager.cc:1504] Loading token signing keys...
I20260812 09:49:28.733182 14858 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 88709457334645bcac18e28a01ca8c12: Generated new TSK 0
I20260812 09:49:28.733922 14858 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 09:49:28.745095 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786527905902985-13708-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786527905902985-13708-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786527905902985-13708-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786527905902985-13708-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.99.1:0
--local_ip_for_outbound_sockets=127.13.99.1
--webserver_interface=127.13.99.1
--webserver_port=0
--tserver_master_addrs=127.13.99.62:43825
--builtin_ntp_servers=127.13.99.20:32897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 09:49:28.875650 14873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:49:28.875900 14873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:49:28.875938 14873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:49:28.879987 14873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:49:28.880043 14873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:49:28.880095 14873 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.99.1
I20260812 09:49:28.884820 14873 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:32897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786527905902985-13708-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786527905902985-13708-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.99.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786527905902985-13708-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.99.1
--webserver_port=0
--tserver_master_addrs=127.13.99.62:43825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14873
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.99.1
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786527905902985-13708-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:49:28.885821 14873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:49:28.887154 14873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:49:28.896590 14888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:49:28.896579 14885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:49:28.896703 14873 server_base.cc:1034] running on GCE node
W20260812 09:49:28.896615 14886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:49:28.897320 14873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:49:28.897928 14873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:49:28.899089 14873 hybrid_clock.cc:648] HybridClock initialized: now 1786528168899077 us; error 24 us; skew 500 ppm
I20260812 09:49:28.900915 14873 webserver.cc:460] Webserver started at http://127.13.99.1:33011/ using document root <none> and password file <none>
I20260812 09:49:28.901496 14873 fs_manager.cc:359] Metadata directory not provided
I20260812 09:49:28.901561 14873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:49:28.901777 14873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:49:28.903805 14873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786527905902985-13708-0/minicluster-data/ts-0/data/instance:
uuid: "03948be528c94976911d0ea5e4b84379"
format_stamp: "Formatted at 2026-08-12 09:49:28 on dist-test-slave-jq4t"
server_key: "263e02f8de81e7c69bbb1545f43d1e4e"
server_key_iv: "6f251846ef8a840e8d7fa574c49194c6"
server_key_version: "encryptionkey@0"
I20260812 09:49:28.904376 14873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786527905902985-13708-0/minicluster-data/ts-0/wal/instance:
uuid: "03948be528c94976911d0ea5e4b84379"
format_stamp: "Formatted at 2026-08-12 09:49:28 on dist-test-slave-jq4t"
server_key: "263e02f8de81e7c69bbb1545f43d1e4e"
server_key_iv: "6f251846ef8a840e8d7fa574c49194c6"
server_key_version: "encryptionkey@0"
I20260812 09:49:28.908082 14873 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260812 09:49:28.910751 14896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:49:28.912070 14873 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260812 09:49:28.912201 14873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786527905902985-13708-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786527905902985-13708-0/minicluster-data/ts-0/wal
uuid: "03948be528c94976911d0ea5e4b84379"
format_stamp: "Formatted at 2026-08-12 09:49:28 on dist-test-slave-jq4t"
server_key: "263e02f8de81e7c69bbb1545f43d1e4e"
server_key_iv: "6f251846ef8a840e8d7fa574c49194c6"
server_key_version: "encryptionkey@0"
I20260812 09:49:28.912307 14873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786527905902985-13708-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786527905902985-13708-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786527905902985-13708-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:49:28.956391 14873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:49:28.957144 14873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:49:28.957352 14873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:49:28.957984 14873 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 09:49:28.958957 14873 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:49:28.959007 14873 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 09:49:28.959049 14873 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:49:28.959102 14873 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 09:49:28.968348 14873 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.1:40809
I20260812 09:49:28.968407 15090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.1:40809 every 8 connection(s)
I20260812 09:49:28.969336 14873 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786527905902985-13708-0/minicluster-data/ts-0/data/info.pb
I20260812 09:49:28.973624 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 14873
I20260812 09:49:28.973778 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786527905902985-13708-0/minicluster-data/ts-0/wal/instance
I20260812 09:49:28.974116 13708 external_mini_cluster.cc:1428] Setting key 0c1428d2f4abcdecb1913f6fde173464
I20260812 09:49:28.976141 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786527905902985-13708-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786527905902985-13708-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786527905902985-13708-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786527905902985-13708-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.99.2:0
--local_ip_for_outbound_sockets=127.13.99.2
--webserver_interface=127.13.99.2
--webserver_port=0
--tserver_master_addrs=127.13.99.62:43825
--builtin_ntp_servers=127.13.99.20:32897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 09:49:28.979437 14836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:28.971296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.1:53607 (local address 127.13.99.62:43825)
0812 09:49:28.971468 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:28.971473 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:28.972643 (+ 1170us) server_negotiation.cc:407] Connection header received
0812 09:49:28.974137 (+ 1494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:28.974142 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:28.974214 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:28.974326 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:49:28.975304 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:28.975848 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:28.977196 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:28.977429 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:28.977545 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:28.978909 (+ 1364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:49:28.978913 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:49:28.978915 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 09:49:28.978929 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:49:28.978950 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 09:49:28.978958 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:49:28.979024 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:28.979230 (+ 206us) server_negotiation.cc:299] Negotiation successful
0812 09:49:28.979288 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260812 09:49:28.980322 15094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:28.971752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:43825 (local address 127.13.99.1:53607)
0812 09:49:28.972429 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:28.972532 (+ 103us) client_negotiation.cc:174] Beginning negotiation
0812 09:49:28.973910 (+ 1378us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:49:28.974473 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:49:28.974480 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:49:28.974566 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:49:28.975155 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:49:28.975169 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:49:28.976003 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:49:28.976007 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:49:28.977061 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:49:28.977072 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:49:28.977588 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:49:28.977593 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:49:28.977831 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:28.978563 (+ 732us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 09:49:28.978566 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:49:28.978625 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 09:49:28.978713 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:49:28.979055 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:49:28.979066 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:49:28.979074 (+ 8us) client_negotiation.cc:769] Sending connection context
0812 09:49:28.979221 (+ 147us) client_negotiation.cc:240] Negotiation successful
0812 09:49:28.979526 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":125,"threads_started":1}
I20260812 09:49:28.981717 15091 heartbeater.cc:347] Connected to a master server at 127.13.99.62:43825
I20260812 09:49:28.982008 15091 heartbeater.cc:464] Registering TS with master...
I20260812 09:49:28.982709 15091 heartbeater.cc:511] Master 127.13.99.62:43825 requested a full tablet report, sending...
I20260812 09:49:28.984443 14766 ts_manager.cc:194] Registered new tserver with Master: 03948be528c94976911d0ea5e4b84379 (127.13.99.1:40809)
I20260812 09:49:28.985693 14766 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.99.1:53607
W20260812 09:49:29.091759 15095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:49:29.092001 15095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:49:29.092038 15095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:49:29.096131 15095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:49:29.096184 15095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:49:29.096266 15095 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.99.2
I20260812 09:49:29.101063 15095 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:32897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786527905902985-13708-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786527905902985-13708-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.99.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786527905902985-13708-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.99.2
--webserver_port=0
--tserver_master_addrs=127.13.99.62:43825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15095
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.99.2
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786527905902985-13708-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:49:29.102089 15095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:49:29.103449 15095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:49:29.112967 15102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:49:29.113039 15095 server_base.cc:1034] running on GCE node
W20260812 09:49:29.113044 15110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:49:29.113194 15105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:49:29.113623 15095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:49:29.114230 15095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:49:29.115399 15095 hybrid_clock.cc:648] HybridClock initialized: now 1786528169115386 us; error 26 us; skew 500 ppm
I20260812 09:49:29.117415 15095 webserver.cc:460] Webserver started at http://127.13.99.2:40773/ using document root <none> and password file <none>
I20260812 09:49:29.117993 15095 fs_manager.cc:359] Metadata directory not provided
I20260812 09:49:29.118055 15095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:49:29.118296 15095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:49:29.120260 15095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786527905902985-13708-0/minicluster-data/ts-1/data/instance:
uuid: "82efc968c6ce4ed7b7774c21105b2f7f"
format_stamp: "Formatted at 2026-08-12 09:49:29 on dist-test-slave-jq4t"
server_key: "2037a38c971a53769e650338701c958d"
server_key_iv: "f86de6d15f09915463e15767df3147dd"
server_key_version: "encryptionkey@0"
I20260812 09:49:29.120878 15095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786527905902985-13708-0/minicluster-data/ts-1/wal/instance:
uuid: "82efc968c6ce4ed7b7774c21105b2f7f"
format_stamp: "Formatted at 2026-08-12 09:49:29 on dist-test-slave-jq4t"
server_key: "2037a38c971a53769e650338701c958d"
server_key_iv: "f86de6d15f09915463e15767df3147dd"
server_key_version: "encryptionkey@0"
I20260812 09:49:29.124295 15095 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260812 09:49:29.126573 15120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:49:29.127731 15095 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 09:49:29.127862 15095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786527905902985-13708-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786527905902985-13708-0/minicluster-data/ts-1/wal
uuid: "82efc968c6ce4ed7b7774c21105b2f7f"
format_stamp: "Formatted at 2026-08-12 09:49:29 on dist-test-slave-jq4t"
server_key: "2037a38c971a53769e650338701c958d"
server_key_iv: "f86de6d15f09915463e15767df3147dd"
server_key_version: "encryptionkey@0"
I20260812 09:49:29.127975 15095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786527905902985-13708-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786527905902985-13708-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786527905902985-13708-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:49:29.149820 15095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:49:29.150455 15095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:49:29.150647 15095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:49:29.151237 15095 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 09:49:29.152176 15095 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:49:29.152251 15095 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 09:49:29.152318 15095 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:49:29.152366 15095 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 09:49:29.161274 15095 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.2:36407
I20260812 09:49:29.161347 15319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.2:36407 every 8 connection(s)
I20260812 09:49:29.162195 15095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786527905902985-13708-0/minicluster-data/ts-1/data/info.pb
I20260812 09:49:29.164002 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 15095
I20260812 09:49:29.164146 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786527905902985-13708-0/minicluster-data/ts-1/wal/instance
I20260812 09:49:29.164542 13708 external_mini_cluster.cc:1428] Setting key 0a1d89a6bd30795cb44f29125a36bfa7
I20260812 09:49:29.166838 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786527905902985-13708-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786527905902985-13708-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786527905902985-13708-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786527905902985-13708-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.99.3:0
--local_ip_for_outbound_sockets=127.13.99.3
--webserver_interface=127.13.99.3
--webserver_port=0
--tserver_master_addrs=127.13.99.62:43825
--builtin_ntp_servers=127.13.99.20:32897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 09:49:29.173240 14836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:29.163984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.2:46739 (local address 127.13.99.62:43825)
0812 09:49:29.164165 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:29.164171 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:29.165528 (+ 1357us) server_negotiation.cc:407] Connection header received
0812 09:49:29.167396 (+ 1868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:29.167399 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:29.167450 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:29.167507 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:49:29.168746 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:29.169258 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:29.170896 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:29.171318 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:29.171445 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:29.172629 (+ 1184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:49:29.172635 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:49:29.172640 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 09:49:29.172657 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:49:29.172683 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 09:49:29.172695 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:49:29.172784 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:29.172950 (+ 166us) server_negotiation.cc:299] Negotiation successful
0812 09:49:29.173024 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 09:49:29.173934 15323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:29.164461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:43825 (local address 127.13.99.2:46739)
0812 09:49:29.165358 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:29.165415 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0812 09:49:29.167178 (+ 1763us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:49:29.167715 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:49:29.167728 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:49:29.167838 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:49:29.168594 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:49:29.168618 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:49:29.169393 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:49:29.169400 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:49:29.170750 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:49:29.170760 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:49:29.171467 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:49:29.171470 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:49:29.171618 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:29.172322 (+ 704us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 09:49:29.172324 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:49:29.172362 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 09:49:29.172424 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:49:29.172785 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:49:29.172793 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:49:29.172798 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 09:49:29.172889 (+ 91us) client_negotiation.cc:240] Negotiation successful
0812 09:49:29.173121 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":174,"threads_started":1}
I20260812 09:49:29.175122 15320 heartbeater.cc:347] Connected to a master server at 127.13.99.62:43825
I20260812 09:49:29.175385 15320 heartbeater.cc:464] Registering TS with master...
I20260812 09:49:29.176052 15320 heartbeater.cc:511] Master 127.13.99.62:43825 requested a full tablet report, sending...
I20260812 09:49:29.177232 14766 ts_manager.cc:194] Registered new tserver with Master: 82efc968c6ce4ed7b7774c21105b2f7f (127.13.99.2:36407)
I20260812 09:49:29.177800 14766 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.99.2:46739
W20260812 09:49:29.284932 15325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:49:29.285172 15325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:49:29.285208 15325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:49:29.289266 15325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:49:29.289318 15325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:49:29.289407 15325 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.99.3
I20260812 09:49:29.294189 15325 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:32897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786527905902985-13708-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786527905902985-13708-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.99.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786527905902985-13708-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.99.3
--webserver_port=0
--tserver_master_addrs=127.13.99.62:43825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15325
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.99.3
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786527905902985-13708-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:49:29.295212 15325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:49:29.296623 15325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:49:29.306334 15340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:49:29.306375 15325 server_base.cc:1034] running on GCE node
W20260812 09:49:29.306246 15335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:49:29.306334 15333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:49:29.306927 15325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:49:29.307583 15325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:49:29.308806 15325 hybrid_clock.cc:648] HybridClock initialized: now 1786528169308778 us; error 41 us; skew 500 ppm
I20260812 09:49:29.311216 15325 webserver.cc:460] Webserver started at http://127.13.99.3:46587/ using document root <none> and password file <none>
I20260812 09:49:29.311806 15325 fs_manager.cc:359] Metadata directory not provided
I20260812 09:49:29.311900 15325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:49:29.312134 15325 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:49:29.314200 15325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786527905902985-13708-0/minicluster-data/ts-2/data/instance:
uuid: "3ef4bb6bcacd41409dcb3b11261dda12"
format_stamp: "Formatted at 2026-08-12 09:49:29 on dist-test-slave-jq4t"
server_key: "ff9b654735447dcc66b145c26ff6d5b1"
server_key_iv: "a0a8a67211011c3ae090b90324cff19c"
server_key_version: "encryptionkey@0"
I20260812 09:49:29.314863 15325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786527905902985-13708-0/minicluster-data/ts-2/wal/instance:
uuid: "3ef4bb6bcacd41409dcb3b11261dda12"
format_stamp: "Formatted at 2026-08-12 09:49:29 on dist-test-slave-jq4t"
server_key: "ff9b654735447dcc66b145c26ff6d5b1"
server_key_iv: "a0a8a67211011c3ae090b90324cff19c"
server_key_version: "encryptionkey@0"
I20260812 09:49:29.318511 15325 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260812 09:49:29.320891 15351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:49:29.322134 15325 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 09:49:29.322280 15325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786527905902985-13708-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786527905902985-13708-0/minicluster-data/ts-2/wal
uuid: "3ef4bb6bcacd41409dcb3b11261dda12"
format_stamp: "Formatted at 2026-08-12 09:49:29 on dist-test-slave-jq4t"
server_key: "ff9b654735447dcc66b145c26ff6d5b1"
server_key_iv: "a0a8a67211011c3ae090b90324cff19c"
server_key_version: "encryptionkey@0"
I20260812 09:49:29.322391 15325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786527905902985-13708-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786527905902985-13708-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786527905902985-13708-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:49:29.338017 15325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:49:29.338644 15325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:49:29.338836 15325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:49:29.339407 15325 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 09:49:29.340313 15325 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:49:29.340387 15325 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 09:49:29.340464 15325 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:49:29.340551 15325 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 09:49:29.349694 15325 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.3:39705
I20260812 09:49:29.349720 15534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.3:39705 every 8 connection(s)
I20260812 09:49:29.350574 15325 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786527905902985-13708-0/minicluster-data/ts-2/data/info.pb
I20260812 09:49:29.355060 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 15325
I20260812 09:49:29.355182 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786527905902985-13708-0/minicluster-data/ts-2/wal/instance
I20260812 09:49:29.355484 13708 external_mini_cluster.cc:1428] Setting key d5b14f6d1f6e57e64c9b6fe845dcff9b
I20260812 09:49:29.359965 14836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:29.352383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.3:59801 (local address 127.13.99.62:43825)
0812 09:49:29.352577 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:29.352581 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:29.353501 (+ 920us) server_negotiation.cc:407] Connection header received
0812 09:49:29.354840 (+ 1339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:29.354843 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:29.354890 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:29.354966 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:49:29.356090 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:29.356767 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:29.358213 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:29.358440 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:29.358531 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:29.359532 (+ 1001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:49:29.359535 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:49:29.359537 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 09:49:29.359550 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:49:29.359569 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 09:49:29.359577 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:49:29.359640 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:29.359779 (+ 139us) server_negotiation.cc:299] Negotiation successful
0812 09:49:29.359829 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260812 09:49:29.360807 15538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:29.352731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:43825 (local address 127.13.99.3:59801)
0812 09:49:29.353371 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:29.353410 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0812 09:49:29.354638 (+ 1228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:49:29.355166 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:49:29.355176 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:49:29.355281 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:49:29.355906 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:49:29.355921 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:49:29.356900 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:49:29.356904 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:49:29.358053 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:49:29.358063 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:49:29.358581 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:49:29.358584 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:49:29.358718 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:29.359288 (+ 570us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 09:49:29.359290 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:49:29.359327 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 09:49:29.359387 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:49:29.359654 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:49:29.359661 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:49:29.359665 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 09:49:29.359755 (+ 90us) client_negotiation.cc:240] Negotiation successful
0812 09:49:29.359976 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":112,"threads_started":1}
I20260812 09:49:29.361964 15535 heartbeater.cc:347] Connected to a master server at 127.13.99.62:43825
I20260812 09:49:29.362259 15535 heartbeater.cc:464] Registering TS with master...
I20260812 09:49:29.362776 15535 heartbeater.cc:511] Master 127.13.99.62:43825 requested a full tablet report, sending...
I20260812 09:49:29.363839 14766 ts_manager.cc:194] Registered new tserver with Master: 3ef4bb6bcacd41409dcb3b11261dda12 (127.13.99.3:39705)
I20260812 09:49:29.364439 14766 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.99.3:59801
I20260812 09:49:29.369328 13708 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 09:49:29.374769 14836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:29.371254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39502 (local address 127.13.99.62:43825)
0812 09:49:29.371389 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:29.371393 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:29.371533 (+ 140us) server_negotiation.cc:407] Connection header received
0812 09:49:29.371624 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:29.371627 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:29.371664 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:29.371725 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:49:29.371939 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:29.372413 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:29.373287 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:29.373499 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:29.373608 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:29.374318 (+ 710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:49:29.374321 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:49:29.374323 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 09:49:29.374334 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:49:29.374349 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 09:49:29.374357 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:49:29.374425 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:29.374523 (+ 98us) server_negotiation.cc:299] Negotiation successful
0812 09:49:29.374569 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260812 09:49:29.377554 14766 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39502:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 09:49:29.381469 14766 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 09:49:29.397374 15557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:29.392096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.3:39705 (local address 127.0.0.1:47442)
0812 09:49:29.392362 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:29.392377 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 09:49:29.392514 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:49:29.393496 (+ 982us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:49:29.393502 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:49:29.393524 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:49:29.393659 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:49:29.393666 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:49:29.394795 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:49:29.394800 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:49:29.395953 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:49:29.395962 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:49:29.396838 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:49:29.396841 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:49:29.396946 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:29.396985 (+ 39us) client_negotiation.cc:769] Sending connection context
0812 09:49:29.397101 (+ 116us) client_negotiation.cc:240] Negotiation successful
0812 09:49:29.397222 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":128,"threads_started":1}
I20260812 09:49:29.397382 15560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:29.392593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47442 (local address 127.13.99.3:39705)
0812 09:49:29.393006 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:29.393019 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:29.393045 (+ 26us) server_negotiation.cc:407] Connection header received
0812 09:49:29.393128 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:29.393135 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:29.393327 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:29.393500 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:49:29.393793 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:29.394676 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:29.396076 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:29.396712 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:29.396836 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:29.397043 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:29.397131 (+ 88us) server_negotiation.cc:299] Negotiation successful
0812 09:49:29.397222 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":185,"threads_started":1}
I20260812 09:49:29.398128 15554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:29.391470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.1:40809 (local address 127.0.0.1:60400)
0812 09:49:29.391817 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:29.391869 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0812 09:49:29.392008 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:49:29.393658 (+ 1650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:49:29.393661 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:49:29.393677 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:49:29.393787 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:49:29.393793 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:49:29.395185 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:49:29.395189 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:49:29.396396 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:49:29.396408 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:49:29.397551 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:49:29.397556 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:49:29.397782 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:29.397803 (+ 21us) client_negotiation.cc:769] Sending connection context
0812 09:49:29.397877 (+ 74us) client_negotiation.cc:240] Negotiation successful
0812 09:49:29.397978 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":76,"thread_start_us":75,"threads_started":1}
I20260812 09:49:29.398144 15559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:29.392183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.2:36407 (local address 127.0.0.1:51274)
0812 09:49:29.392517 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:29.392531 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 09:49:29.392628 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:49:29.393898 (+ 1270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:49:29.393901 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:49:29.393917 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:49:29.393985 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:49:29.393990 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:49:29.395299 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:49:29.395302 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:49:29.396499 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:49:29.396510 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:49:29.397551 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:49:29.397556 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:49:29.397807 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:29.397842 (+ 35us) client_negotiation.cc:769] Sending connection context
0812 09:49:29.397915 (+ 73us) client_negotiation.cc:240] Negotiation successful
0812 09:49:29.397991 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"mutex_wait_us":64,"thread_start_us":147,"threads_started":1}
I20260812 09:49:29.398223 15561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:29.392831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51274 (local address 127.13.99.2:36407)
0812 09:49:29.393222 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:29.393234 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:29.393252 (+ 18us) server_negotiation.cc:407] Connection header received
0812 09:49:29.393315 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:29.393320 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:29.393480 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:29.393684 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:49:29.394112 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:29.395188 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:29.396643 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:29.397399 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:29.397607 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:29.397800 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:29.397965 (+ 165us) server_negotiation.cc:299] Negotiation successful
0812 09:49:29.398059 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":117,"threads_started":1}
I20260812 09:49:29.398895 15555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:29.391627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60400 (local address 127.13.99.1:40809)
0812 09:49:29.393155 (+ 1528us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:29.393167 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:29.393192 (+ 25us) server_negotiation.cc:407] Connection header received
0812 09:49:29.393278 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:29.393283 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:29.393480 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:29.393684 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:49:29.393885 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:29.395043 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:29.396590 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:29.397399 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:29.398463 (+ 1064us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:29.398604 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:29.398643 (+ 39us) server_negotiation.cc:299] Negotiation successful
0812 09:49:29.398712 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1406,"thread_start_us":111,"threads_started":1}
I20260812 09:49:29.401049 15207 tablet_service.cc:1467] Processing CreateTablet for tablet e7def00295c446249a5aab7410abcee9 (DEFAULT_TABLE table=test-table [id=03bb9542ee9243e9a2f4a91dbb70b946]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 09:49:29.401096 15429 tablet_service.cc:1467] Processing CreateTablet for tablet e7def00295c446249a5aab7410abcee9 (DEFAULT_TABLE table=test-table [id=03bb9542ee9243e9a2f4a91dbb70b946]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 09:49:29.401350 14973 tablet_service.cc:1467] Processing CreateTablet for tablet e7def00295c446249a5aab7410abcee9 (DEFAULT_TABLE table=test-table [id=03bb9542ee9243e9a2f4a91dbb70b946]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 09:49:29.402120 15207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7def00295c446249a5aab7410abcee9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:49:29.402170 14973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7def00295c446249a5aab7410abcee9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:49:29.402164 15429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7def00295c446249a5aab7410abcee9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:49:29.408599 15562 tablet_bootstrap.cc:492] T e7def00295c446249a5aab7410abcee9 P 82efc968c6ce4ed7b7774c21105b2f7f: Bootstrap starting.
I20260812 09:49:29.409788 15564 tablet_bootstrap.cc:492] T e7def00295c446249a5aab7410abcee9 P 03948be528c94976911d0ea5e4b84379: Bootstrap starting.
I20260812 09:49:29.410159 15563 tablet_bootstrap.cc:492] T e7def00295c446249a5aab7410abcee9 P 3ef4bb6bcacd41409dcb3b11261dda12: Bootstrap starting.
I20260812 09:49:29.411240 15562 tablet_bootstrap.cc:654] T e7def00295c446249a5aab7410abcee9 P 82efc968c6ce4ed7b7774c21105b2f7f: Neither blocks nor log segments found. Creating new log.
I20260812 09:49:29.411954 15564 tablet_bootstrap.cc:654] T e7def00295c446249a5aab7410abcee9 P 03948be528c94976911d0ea5e4b84379: Neither blocks nor log segments found. Creating new log.
I20260812 09:49:29.411975 15562 log.cc:824] T e7def00295c446249a5aab7410abcee9 P 82efc968c6ce4ed7b7774c21105b2f7f: Log is configured to *not* fsync() on all Append() calls
I20260812 09:49:29.412174 15563 tablet_bootstrap.cc:654] T e7def00295c446249a5aab7410abcee9 P 3ef4bb6bcacd41409dcb3b11261dda12: Neither blocks nor log segments found. Creating new log.
I20260812 09:49:29.412843 15564 log.cc:824] T e7def00295c446249a5aab7410abcee9 P 03948be528c94976911d0ea5e4b84379: Log is configured to *not* fsync() on all Append() calls
I20260812 09:49:29.412917 15563 log.cc:824] T e7def00295c446249a5aab7410abcee9 P 3ef4bb6bcacd41409dcb3b11261dda12: Log is configured to *not* fsync() on all Append() calls
I20260812 09:49:29.414126 15562 tablet_bootstrap.cc:492] T e7def00295c446249a5aab7410abcee9 P 82efc968c6ce4ed7b7774c21105b2f7f: No bootstrap required, opened a new log
I20260812 09:49:29.414337 15562 ts_tablet_manager.cc:1397] T e7def00295c446249a5aab7410abcee9 P 82efc968c6ce4ed7b7774c21105b2f7f: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260812 09:49:29.414835 15564 tablet_bootstrap.cc:492] T e7def00295c446249a5aab7410abcee9 P 03948be528c94976911d0ea5e4b84379: No bootstrap required, opened a new log
I20260812 09:49:29.414978 15563 tablet_bootstrap.cc:492] T e7def00295c446249a5aab7410abcee9 P 3ef4bb6bcacd41409dcb3b11261dda12: No bootstrap required, opened a new log
I20260812 09:49:29.415057 15564 ts_tablet_manager.cc:1397] T e7def00295c446249a5aab7410abcee9 P 03948be528c94976911d0ea5e4b84379: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260812 09:49:29.415138 15563 ts_tablet_manager.cc:1397] T e7def00295c446249a5aab7410abcee9 P 3ef4bb6bcacd41409dcb3b11261dda12: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260812 09:49:29.418078 15562 raft_consensus.cc:348] T e7def00295c446249a5aab7410abcee9 P 82efc968c6ce4ed7b7774c21105b2f7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03948be528c94976911d0ea5e4b84379" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 40809 } } peers { permanent_uuid: "82efc968c6ce4ed7b7774c21105b2f7f" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 36407 } } peers { permanent_uuid: "3ef4bb6bcacd41409dcb3b11261dda12" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 39705 } }
I20260812 09:49:29.418323 15562 raft_consensus.cc:374] T e7def00295c446249a5aab7410abcee9 P 82efc968c6ce4ed7b7774c21105b2f7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:49:29.418378 15562 raft_consensus.cc:729] T e7def00295c446249a5aab7410abcee9 P 82efc968c6ce4ed7b7774c21105b2f7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82efc968c6ce4ed7b7774c21105b2f7f, State: Initialized, Role: FOLLOWER
I20260812 09:49:29.418823 15562 consensus_queue.cc:260] T e7def00295c446249a5aab7410abcee9 P 82efc968c6ce4ed7b7774c21105b2f7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03948be528c94976911d0ea5e4b84379" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 40809 } } peers { permanent_uuid: "82efc968c6ce4ed7b7774c21105b2f7f" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 36407 } } peers { permanent_uuid: "3ef4bb6bcacd41409dcb3b11261dda12" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 39705 } }
I20260812 09:49:29.419551 15320 heartbeater.cc:503] Master 127.13.99.62:43825 was elected leader, sending a full tablet report...
I20260812 09:49:29.419466 15563 raft_consensus.cc:348] T e7def00295c446249a5aab7410abcee9 P 3ef4bb6bcacd41409dcb3b11261dda12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03948be528c94976911d0ea5e4b84379" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 40809 } } peers { permanent_uuid: "82efc968c6ce4ed7b7774c21105b2f7f" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 36407 } } peers { permanent_uuid: "3ef4bb6bcacd41409dcb3b11261dda12" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 39705 } }
I20260812 09:49:29.419718 15563 raft_consensus.cc:374] T e7def00295c446249a5aab7410abcee9 P 3ef4bb6bcacd41409dcb3b11261dda12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:49:29.419785 15563 raft_consensus.cc:729] T e7def00295c446249a5aab7410abcee9 P 3ef4bb6bcacd41409dcb3b11261dda12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ef4bb6bcacd41409dcb3b11261dda12, State: Initialized, Role: FOLLOWER
I20260812 09:49:29.419935 15562 ts_tablet_manager.cc:1428] T e7def00295c446249a5aab7410abcee9 P 82efc968c6ce4ed7b7774c21105b2f7f: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260812 09:49:29.420130 15564 raft_consensus.cc:348] T e7def00295c446249a5aab7410abcee9 P 03948be528c94976911d0ea5e4b84379 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03948be528c94976911d0ea5e4b84379" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 40809 } } peers { permanent_uuid: "82efc968c6ce4ed7b7774c21105b2f7f" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 36407 } } peers { permanent_uuid: "3ef4bb6bcacd41409dcb3b11261dda12" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 39705 } }
I20260812 09:49:29.420248 15563 consensus_queue.cc:260] T e7def00295c446249a5aab7410abcee9 P 3ef4bb6bcacd41409dcb3b11261dda12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03948be528c94976911d0ea5e4b84379" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 40809 } } peers { permanent_uuid: "82efc968c6ce4ed7b7774c21105b2f7f" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 36407 } } peers { permanent_uuid: "3ef4bb6bcacd41409dcb3b11261dda12" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 39705 } }
I20260812 09:49:29.420370 15564 raft_consensus.cc:374] T e7def00295c446249a5aab7410abcee9 P 03948be528c94976911d0ea5e4b84379 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:49:29.420430 15564 raft_consensus.cc:729] T e7def00295c446249a5aab7410abcee9 P 03948be528c94976911d0ea5e4b84379 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03948be528c94976911d0ea5e4b84379, State: Initialized, Role: FOLLOWER
I20260812 09:49:29.420909 15564 consensus_queue.cc:260] T e7def00295c446249a5aab7410abcee9 P 03948be528c94976911d0ea5e4b84379 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03948be528c94976911d0ea5e4b84379" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 40809 } } peers { permanent_uuid: "82efc968c6ce4ed7b7774c21105b2f7f" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 36407 } } peers { permanent_uuid: "3ef4bb6bcacd41409dcb3b11261dda12" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 39705 } }
I20260812 09:49:29.421190 15563 ts_tablet_manager.cc:1428] T e7def00295c446249a5aab7410abcee9 P 3ef4bb6bcacd41409dcb3b11261dda12: Time spent starting tablet: real 0.006s user 0.000s sys 0.005s
I20260812 09:49:29.421550 15535 heartbeater.cc:503] Master 127.13.99.62:43825 was elected leader, sending a full tablet report...
I20260812 09:49:29.421882 15564 ts_tablet_manager.cc:1428] T e7def00295c446249a5aab7410abcee9 P 03948be528c94976911d0ea5e4b84379: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260812 09:49:29.422255 15091 heartbeater.cc:503] Master 127.13.99.62:43825 was elected leader, sending a full tablet report...
W20260812 09:49:29.470758 15093 tablet.cc:2369] T e7def00295c446249a5aab7410abcee9 P 03948be528c94976911d0ea5e4b84379: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 09:49:29.549618 15573 raft_consensus.cc:482] T e7def00295c446249a5aab7410abcee9 P 03948be528c94976911d0ea5e4b84379 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 09:49:29.549881 15573 raft_consensus.cc:504] T e7def00295c446249a5aab7410abcee9 P 03948be528c94976911d0ea5e4b84379 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03948be528c94976911d0ea5e4b84379" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 40809 } } peers { permanent_uuid: "82efc968c6ce4ed7b7774c21105b2f7f" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 36407 } } peers { permanent_uuid: "3ef4bb6bcacd41409dcb3b11261dda12" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 39705 } }
I20260812 09:49:29.551208 15573 leader_election.cc:288] T e7def00295c446249a5aab7410abcee9 P 03948be528c94976911d0ea5e4b84379 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82efc968c6ce4ed7b7774c21105b2f7f (127.13.99.2:36407), 3ef4bb6bcacd41409dcb3b11261dda12 (127.13.99.3:39705)
I20260812 09:49:29.555339 15561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:29.551577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.1:60841 (local address 127.13.99.2:36407)
0812 09:49:29.551801 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:29.551805 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:29.551822 (+ 17us) server_negotiation.cc:407] Connection header received
0812 09:49:29.551998 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:29.552002 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:29.552073 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:29.552155 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:49:29.552530 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:29.553125 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:29.554292 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:29.554699 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:29.554782 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:29.554858 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:29.555124 (+ 266us) server_negotiation.cc:299] Negotiation successful
0812 09:49:29.555200 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
I20260812 09:49:29.555353 15560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:29.551829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.99.1:56125 (local address 127.13.99.3:39705)
0812 09:49:29.551990 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:29.552001 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:29.552097 (+ 96us) server_negotiation.cc:407] Connection header received
0812 09:49:29.552235 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:29.552238 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:29.552287 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:29.552369 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 09:49:29.552715 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:29.553237 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:29.554346 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:29.554782 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:29.554889 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:29.554951 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:29.555171 (+ 220us) server_negotiation.cc:299] Negotiation successful
0812 09:49:29.555249 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260812 09:49:29.555365 15576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:29.551574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.3:39705 (local address 127.13.99.1:56125)
0812 09:49:29.552001 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:29.552015 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 09:49:29.552126 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:49:29.552393 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:49:29.552395 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:49:29.552411 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:49:29.552596 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:49:29.552603 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:49:29.553348 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:49:29.553351 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:49:29.554188 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:49:29.554198 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:49:29.554942 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:49:29.554945 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:49:29.555018 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:29.555037 (+ 19us) client_negotiation.cc:769] Sending connection context
0812 09:49:29.555135 (+ 98us) client_negotiation.cc:240] Negotiation successful
0812 09:49:29.555200 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"mutex_wait_us":22,"thread_start_us":121,"threads_started":1}
I20260812 09:49:29.555371 15575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:29.551411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.2:36407 (local address 127.13.99.1:60841)
0812 09:49:29.551723 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:29.551741 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 09:49:29.551857 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:49:29.552206 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:49:29.552209 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:49:29.552231 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 09:49:29.552386 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:49:29.552393 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:49:29.553245 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:49:29.553249 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:49:29.554159 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:49:29.554169 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:49:29.554831 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:49:29.554838 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:49:29.554941 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:29.554999 (+ 58us) client_negotiation.cc:769] Sending connection context
0812 09:49:29.555073 (+ 74us) client_negotiation.cc:240] Negotiation successful
0812 09:49:29.555159 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"mutex_wait_us":32,"thread_start_us":146,"threads_started":1}
I20260812 09:49:29.556080 15463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7def00295c446249a5aab7410abcee9" candidate_uuid: "03948be528c94976911d0ea5e4b84379" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ef4bb6bcacd41409dcb3b11261dda12" is_pre_election: true
I20260812 09:49:29.556402 15463 raft_consensus.cc:2454] T e7def00295c446249a5aab7410abcee9 P 3ef4bb6bcacd41409dcb3b11261dda12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03948be528c94976911d0ea5e4b84379 in term 0.
I20260812 09:49:29.556437 15234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7def00295c446249a5aab7410abcee9" candidate_uuid: "03948be528c94976911d0ea5e4b84379" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82efc968c6ce4ed7b7774c21105b2f7f" is_pre_election: true
I20260812 09:49:29.556792 15234 raft_consensus.cc:2454] T e7def00295c446249a5aab7410abcee9 P 82efc968c6ce4ed7b7774c21105b2f7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03948be528c94976911d0ea5e4b84379 in term 0.
I20260812 09:49:29.556967 14903 leader_election.cc:302] T e7def00295c446249a5aab7410abcee9 P 03948be528c94976911d0ea5e4b84379 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03948be528c94976911d0ea5e4b84379, 3ef4bb6bcacd41409dcb3b11261dda12; no voters:
I20260812 09:49:29.557276 15573 raft_consensus.cc:2781] T e7def00295c446249a5aab7410abcee9 P 03948be528c94976911d0ea5e4b84379 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 09:49:29.557369 15573 raft_consensus.cc:482] T e7def00295c446249a5aab7410abcee9 P 03948be528c94976911d0ea5e4b84379 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 09:49:29.557404 15573 raft_consensus.cc:3037] T e7def00295c446249a5aab7410abcee9 P 03948be528c94976911d0ea5e4b84379 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:49:29.558538 15573 raft_consensus.cc:504] T e7def00295c446249a5aab7410abcee9 P 03948be528c94976911d0ea5e4b84379 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03948be528c94976911d0ea5e4b84379" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 40809 } } peers { permanent_uuid: "82efc968c6ce4ed7b7774c21105b2f7f" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 36407 } } peers { permanent_uuid: "3ef4bb6bcacd41409dcb3b11261dda12" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 39705 } }
I20260812 09:49:29.558943 15573 leader_election.cc:288] T e7def00295c446249a5aab7410abcee9 P 03948be528c94976911d0ea5e4b84379 [CANDIDATE]: Term 1 election: Requested vote from peers 82efc968c6ce4ed7b7774c21105b2f7f (127.13.99.2:36407), 3ef4bb6bcacd41409dcb3b11261dda12 (127.13.99.3:39705)
I20260812 09:49:29.559301 15234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7def00295c446249a5aab7410abcee9" candidate_uuid: "03948be528c94976911d0ea5e4b84379" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82efc968c6ce4ed7b7774c21105b2f7f"
I20260812 09:49:29.559338 15463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7def00295c446249a5aab7410abcee9" candidate_uuid: "03948be528c94976911d0ea5e4b84379" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ef4bb6bcacd41409dcb3b11261dda12"
I20260812 09:49:29.559425 15234 raft_consensus.cc:3037] T e7def00295c446249a5aab7410abcee9 P 82efc968c6ce4ed7b7774c21105b2f7f [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:49:29.559499 15463 raft_consensus.cc:3037] T e7def00295c446249a5aab7410abcee9 P 3ef4bb6bcacd41409dcb3b11261dda12 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:49:29.560554 15234 raft_consensus.cc:2454] T e7def00295c446249a5aab7410abcee9 P 82efc968c6ce4ed7b7774c21105b2f7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03948be528c94976911d0ea5e4b84379 in term 1.
I20260812 09:49:29.560690 15463 raft_consensus.cc:2454] T e7def00295c446249a5aab7410abcee9 P 3ef4bb6bcacd41409dcb3b11261dda12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03948be528c94976911d0ea5e4b84379 in term 1.
I20260812 09:49:29.560928 14901 leader_election.cc:302] T e7def00295c446249a5aab7410abcee9 P 03948be528c94976911d0ea5e4b84379 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03948be528c94976911d0ea5e4b84379, 82efc968c6ce4ed7b7774c21105b2f7f; no voters:
I20260812 09:49:29.561127 15573 raft_consensus.cc:2781] T e7def00295c446249a5aab7410abcee9 P 03948be528c94976911d0ea5e4b84379 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 09:49:29.561395 15573 raft_consensus.cc:686] T e7def00295c446249a5aab7410abcee9 P 03948be528c94976911d0ea5e4b84379 [term 1 LEADER]: Becoming Leader. State: Replica: 03948be528c94976911d0ea5e4b84379, State: Running, Role: LEADER
I20260812 09:49:29.561725 15573 consensus_queue.cc:237] T e7def00295c446249a5aab7410abcee9 P 03948be528c94976911d0ea5e4b84379 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03948be528c94976911d0ea5e4b84379" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 40809 } } peers { permanent_uuid: "82efc968c6ce4ed7b7774c21105b2f7f" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 36407 } } peers { permanent_uuid: "3ef4bb6bcacd41409dcb3b11261dda12" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 39705 } }
I20260812 09:49:29.565284 14764 catalog_manager.cc:5495] T e7def00295c446249a5aab7410abcee9 P 03948be528c94976911d0ea5e4b84379 reported cstate change: term changed from 0 to 1, leader changed from <none> to 03948be528c94976911d0ea5e4b84379 (127.13.99.1). New cstate: current_term: 1 leader_uuid: "03948be528c94976911d0ea5e4b84379" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03948be528c94976911d0ea5e4b84379" member_type: VOTER last_known_addr { host: "127.13.99.1" port: 40809 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82efc968c6ce4ed7b7774c21105b2f7f" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 36407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ef4bb6bcacd41409dcb3b11261dda12" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 39705 } health_report { overall_health: UNKNOWN } } }
I20260812 09:49:29.585795 15555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:29.581735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60406 (local address 127.13.99.1:40809)
0812 09:49:29.581877 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:29.581881 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:29.581895 (+ 14us) server_negotiation.cc:407] Connection header received
0812 09:49:29.582063 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:29.582065 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:29.582124 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:29.582246 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 09:49:29.582494 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:29.583001 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:29.584192 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:29.584411 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:29.584573 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:29.584903 (+ 330us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 09:49:29.585403 (+ 500us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 09:49:29.585531 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:29.585594 (+ 63us) server_negotiation.cc:299] Negotiation successful
0812 09:49:29.585650 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260812 09:49:29.592180 15234 raft_consensus.cc:1261] T e7def00295c446249a5aab7410abcee9 P 82efc968c6ce4ed7b7774c21105b2f7f [term 1 FOLLOWER]: Refusing update from remote peer 03948be528c94976911d0ea5e4b84379: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 09:49:29.592182 15463 raft_consensus.cc:1261] T e7def00295c446249a5aab7410abcee9 P 3ef4bb6bcacd41409dcb3b11261dda12 [term 1 FOLLOWER]: Refusing update from remote peer 03948be528c94976911d0ea5e4b84379: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 09:49:29.593037 15578 consensus_queue.cc:1035] T e7def00295c446249a5aab7410abcee9 P 03948be528c94976911d0ea5e4b84379 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82efc968c6ce4ed7b7774c21105b2f7f" member_type: VOTER last_known_addr { host: "127.13.99.2" port: 36407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 09:49:29.593250 15573 consensus_queue.cc:1035] T e7def00295c446249a5aab7410abcee9 P 03948be528c94976911d0ea5e4b84379 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ef4bb6bcacd41409dcb3b11261dda12" member_type: VOTER last_known_addr { host: "127.13.99.3" port: 39705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 09:49:29.611505 14764 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39502:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 09:49:29.611696 14764 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39502:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 09:49:29.614248 14764 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 88709457334645bcac18e28a01ca8c12: Sending DeleteTablet for 3 replicas of tablet e7def00295c446249a5aab7410abcee9
I20260812 09:49:29.615204 15207 tablet_service.cc:1514] Processing DeleteTablet for tablet e7def00295c446249a5aab7410abcee9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 09:49:29 UTC) from {username='slave'} at 127.0.0.1:51274
I20260812 09:49:29.615216 15429 tablet_service.cc:1514] Processing DeleteTablet for tablet e7def00295c446249a5aab7410abcee9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 09:49:29 UTC) from {username='slave'} at 127.0.0.1:47442
I20260812 09:49:29.615744 15600 tablet_replica.cc:330] T e7def00295c446249a5aab7410abcee9 P 3ef4bb6bcacd41409dcb3b11261dda12: stopping tablet replica
I20260812 09:49:29.615764 15601 tablet_replica.cc:330] T e7def00295c446249a5aab7410abcee9 P 82efc968c6ce4ed7b7774c21105b2f7f: stopping tablet replica
I20260812 09:49:29.616093 15600 raft_consensus.cc:2229] T e7def00295c446249a5aab7410abcee9 P 3ef4bb6bcacd41409dcb3b11261dda12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 09:49:29.616247 15601 raft_consensus.cc:2229] T e7def00295c446249a5aab7410abcee9 P 82efc968c6ce4ed7b7774c21105b2f7f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 09:49:29.616427 15600 raft_consensus.cc:2258] T e7def00295c446249a5aab7410abcee9 P 3ef4bb6bcacd41409dcb3b11261dda12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 09:49:29.616592 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 14873
I20260812 09:49:29.619601 15601 raft_consensus.cc:2258] T e7def00295c446249a5aab7410abcee9 P 82efc968c6ce4ed7b7774c21105b2f7f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 09:49:29.624398 15601 ts_tablet_manager.cc:1905] T e7def00295c446249a5aab7410abcee9 P 82efc968c6ce4ed7b7774c21105b2f7f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 09:49:29.627686 15600 ts_tablet_manager.cc:1905] T e7def00295c446249a5aab7410abcee9 P 3ef4bb6bcacd41409dcb3b11261dda12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 09:49:29.628824 15601 ts_tablet_manager.cc:1918] T e7def00295c446249a5aab7410abcee9 P 82efc968c6ce4ed7b7774c21105b2f7f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 09:49:29.629001 15601 log.cc:1196] T e7def00295c446249a5aab7410abcee9 P 82efc968c6ce4ed7b7774c21105b2f7f: Deleting WAL directory at /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786527905902985-13708-0/minicluster-data/ts-1/wal/wals/e7def00295c446249a5aab7410abcee9
I20260812 09:49:29.629355 15601 ts_tablet_manager.cc:1939] T e7def00295c446249a5aab7410abcee9 P 82efc968c6ce4ed7b7774c21105b2f7f: Deleting consensus metadata
I20260812 09:49:29.630834 14740 catalog_manager.cc:4841] TS 82efc968c6ce4ed7b7774c21105b2f7f (127.13.99.2:36407): tablet e7def00295c446249a5aab7410abcee9 (table test-table [id=03bb9542ee9243e9a2f4a91dbb70b946]) successfully deleted
I20260812 09:49:29.631456 15600 ts_tablet_manager.cc:1918] T e7def00295c446249a5aab7410abcee9 P 3ef4bb6bcacd41409dcb3b11261dda12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 09:49:29.631568 15600 log.cc:1196] T e7def00295c446249a5aab7410abcee9 P 3ef4bb6bcacd41409dcb3b11261dda12: Deleting WAL directory at /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786527905902985-13708-0/minicluster-data/ts-2/wal/wals/e7def00295c446249a5aab7410abcee9
I20260812 09:49:29.631909 15600 ts_tablet_manager.cc:1939] T e7def00295c446249a5aab7410abcee9 P 3ef4bb6bcacd41409dcb3b11261dda12: Deleting consensus metadata
I20260812 09:49:29.633279 14741 catalog_manager.cc:4841] TS 3ef4bb6bcacd41409dcb3b11261dda12 (127.13.99.3:39705): tablet e7def00295c446249a5aab7410abcee9 (table test-table [id=03bb9542ee9243e9a2f4a91dbb70b946]) successfully deleted
W20260812 09:49:29.637048 14740 connection.cc:535] client connection to 127.13.99.1:40809 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260812 09:49:29.637209 14740 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 09:49:29.637337 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 15095
I20260812 09:49:29.647904 15559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:29.647519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.1:40809 (local address 127.0.0.1:60414)
0812 09:49:29.647721 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0812 09:49:29.647796 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.1:40809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
W20260812 09:49:29.648132 14740 catalog_manager.cc:4584] TS 03948be528c94976911d0ea5e4b84379 (127.13.99.1:40809): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e7def00295c446249a5aab7410abcee9: Network error: Client connection negotiation failed: client connection to 127.13.99.1:40809: connect: Connection refused (error 111)
I20260812 09:49:29.654950 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 15325
I20260812 09:49:29.666169 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 14719
2026-08-12T09:49:29Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1196 ms)
[----------] 6 tests from EncryptionPolicyTest (11928 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786527905902985-13708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 09:49:29 dist-test-slave-jq4t krb5kdc[15608](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 09:49:29 dist-test-slave-jq4t krb5kdc[15608](info): set up 2 sockets
Aug 12 09:49:29 dist-test-slave-jq4t krb5kdc[15608](info): commencing operation
krb5kdc: starting...
W20260812 09:49:30.958063 13708 mini_kdc.cc:121] Time spent starting KDC: real 1.265s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786527905902985-13708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786527905902985-13708-0/krb5kdc/test-user.keytab.
Aug 12 09:49:31 dist-test-slave-jq4t krb5kdc[15608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T09:49:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T09:49:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786527905902985-13708-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786527905902985-13708-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 09:49:31.087226 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786527905902985-13708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786527905902985-13708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786527905902985-13708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34403
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786527905902985-13708-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34403
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:49:31.199350 15638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:49:31.199597 15638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:49:31.199638 15638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:49:31.203476 15638 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 09:49:31.203534 15638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:49:31.203553 15638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:49:31.203569 15638 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 09:49:31.203584 15638 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 09:49:31.208662 15638 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786527905902985-13708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786527905902985-13708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34403
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786527905902985-13708-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34403
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15638
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786527905902985-13708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:49:31.209640 15638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:49:31.210606 15638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:49:31.219201 15646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:49:31.219209 15643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:49:31.219281 15644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:49:31.219383 15638 server_base.cc:1034] running on GCE node
I20260812 09:49:31.220153 15638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:49:31.221290 15638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:49:31.222453 15638 hybrid_clock.cc:648] HybridClock initialized: now 1786528171222434 us; error 33 us; skew 500 ppm
Aug 12 09:49:31 dist-test-slave-jq4t krb5kdc[15608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528171, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:49:31.225198 15638 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 09:49:31.226017 15638 webserver.cc:460] Webserver started at http://127.0.0.1:36037/ using document root <none> and password file <none>
I20260812 09:49:31.226542 15638 fs_manager.cc:359] Metadata directory not provided
I20260812 09:49:31.226601 15638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:49:31.226765 15638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:49:31.228799 15638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786527905902985-13708-0/minicluster-data/master-0/data/instance:
uuid: "abf4cd720e4b47599738c8162f951a5e"
format_stamp: "Formatted at 2026-08-12 09:49:31 on dist-test-slave-jq4t"
server_key: "3b17c72d2fb138462f9c6bc1e9705a89"
server_key_iv: "e84d9886afe139b0feeac72a72b468e9"
server_key_version: "encryptionkey@0"
I20260812 09:49:31.229318 15638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786527905902985-13708-0/minicluster-data/master-0/wal/instance:
uuid: "abf4cd720e4b47599738c8162f951a5e"
format_stamp: "Formatted at 2026-08-12 09:49:31 on dist-test-slave-jq4t"
server_key: "3b17c72d2fb138462f9c6bc1e9705a89"
server_key_iv: "e84d9886afe139b0feeac72a72b468e9"
server_key_version: "encryptionkey@0"
I20260812 09:49:31.232975 15638 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260812 09:49:31.235453 15656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:49:31.236918 15638 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 09:49:31.237026 15638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786527905902985-13708-0/minicluster-data/master-0/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786527905902985-13708-0/minicluster-data/master-0/wal
uuid: "abf4cd720e4b47599738c8162f951a5e"
format_stamp: "Formatted at 2026-08-12 09:49:31 on dist-test-slave-jq4t"
server_key: "3b17c72d2fb138462f9c6bc1e9705a89"
server_key_iv: "e84d9886afe139b0feeac72a72b468e9"
server_key_version: "encryptionkey@0"
I20260812 09:49:31.237115 15638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786527905902985-13708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786527905902985-13708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786527905902985-13708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:49:31.258414 15638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:49:31.261466 15638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:49:31.261626 15638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:49:31.268824 15638 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34403
I20260812 09:49:31.268838 15737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34403 every 8 connection(s)
I20260812 09:49:31.269794 15638 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
I20260812 09:49:31.272599 15740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:49:31.273710 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 15638
I20260812 09:49:31.273875 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786527905902985-13708-0/minicluster-data/master-0/wal/instance
I20260812 09:49:31.274179 13708 external_mini_cluster.cc:1428] Setting key 113ded07059b126c05b641ebc35a70a3
I20260812 09:49:31.279007 15740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abf4cd720e4b47599738c8162f951a5e: Bootstrap starting.
I20260812 09:49:31.281774 15740 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P abf4cd720e4b47599738c8162f951a5e: Neither blocks nor log segments found. Creating new log.
I20260812 09:49:31.282728 15740 log.cc:824] T 00000000000000000000000000000000 P abf4cd720e4b47599738c8162f951a5e: Log is configured to *not* fsync() on all Append() calls
Aug 12 09:49:31 dist-test-slave-jq4t krb5kdc[15608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260812 09:49:31.285024 15740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abf4cd720e4b47599738c8162f951a5e: No bootstrap required, opened a new log
I20260812 09:49:31.289404 15740 raft_consensus.cc:348] T 00000000000000000000000000000000 P abf4cd720e4b47599738c8162f951a5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abf4cd720e4b47599738c8162f951a5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34403 } }
I20260812 09:49:31.289628 15740 raft_consensus.cc:374] T 00000000000000000000000000000000 P abf4cd720e4b47599738c8162f951a5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:49:31.289688 15740 raft_consensus.cc:729] T 00000000000000000000000000000000 P abf4cd720e4b47599738c8162f951a5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abf4cd720e4b47599738c8162f951a5e, State: Initialized, Role: FOLLOWER
I20260812 09:49:31.289727 15744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:31.275507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55429 (local address 127.0.0.1:34403)
0812 09:49:31.275901 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:31.275917 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:31.275952 (+ 35us) server_negotiation.cc:407] Connection header received
0812 09:49:31.276664 (+ 712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:31.276685 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:31.277012 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:31.277280 (+ 268us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:49:31.278247 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:31.279320 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:31.280471 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:31.280987 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:31.281112 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:31.284041 (+ 2929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:49:31.284046 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:49:31.284059 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:49:31.284083 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:49:31.286644 (+ 2561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:49:31.287291 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:49:31.287296 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:49:31.287301 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:49:31.287410 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:49:31.287803 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:49:31.287806 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:49:31.287807 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:49:31.288185 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:49:31.288343 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:31.288722 (+ 379us) server_negotiation.cc:299] Negotiation successful
0812 09:49:31.288901 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":128,"threads_started":1}
I20260812 09:49:31.290187 15740 consensus_queue.cc:260] T 00000000000000000000000000000000 P abf4cd720e4b47599738c8162f951a5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abf4cd720e4b47599738c8162f951a5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34403 } }
I20260812 09:49:31.290318 15740 raft_consensus.cc:388] T 00000000000000000000000000000000 P abf4cd720e4b47599738c8162f951a5e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 09:49:31.290401 15740 raft_consensus.cc:482] T 00000000000000000000000000000000 P abf4cd720e4b47599738c8162f951a5e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 09:49:31.290532 15740 raft_consensus.cc:3037] T 00000000000000000000000000000000 P abf4cd720e4b47599738c8162f951a5e [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:49:31.291760 15740 raft_consensus.cc:504] T 00000000000000000000000000000000 P abf4cd720e4b47599738c8162f951a5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abf4cd720e4b47599738c8162f951a5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34403 } }
I20260812 09:49:31.292168 15740 leader_election.cc:302] T 00000000000000000000000000000000 P abf4cd720e4b47599738c8162f951a5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: abf4cd720e4b47599738c8162f951a5e; no voters:
I20260812 09:49:31.292603 15740 leader_election.cc:288] T 00000000000000000000000000000000 P abf4cd720e4b47599738c8162f951a5e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 09:49:31.292696 15750 raft_consensus.cc:2781] T 00000000000000000000000000000000 P abf4cd720e4b47599738c8162f951a5e [term 1 FOLLOWER]: Leader election won for term 1
I20260812 09:49:31.292906 15750 raft_consensus.cc:686] T 00000000000000000000000000000000 P abf4cd720e4b47599738c8162f951a5e [term 1 LEADER]: Becoming Leader. State: Replica: abf4cd720e4b47599738c8162f951a5e, State: Running, Role: LEADER
I20260812 09:49:31.293078 13708 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 09:49:31.293233 15750 consensus_queue.cc:237] T 00000000000000000000000000000000 P abf4cd720e4b47599738c8162f951a5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abf4cd720e4b47599738c8162f951a5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34403 } }
I20260812 09:49:31.293965 15740 sys_catalog.cc:564] T 00000000000000000000000000000000 P abf4cd720e4b47599738c8162f951a5e [sys.catalog]: configured and running, proceeding with master startup.
I20260812 09:49:31.294337 15753 sys_catalog.cc:455] T 00000000000000000000000000000000 P abf4cd720e4b47599738c8162f951a5e [sys.catalog]: SysCatalogTable state changed. Reason: New leader abf4cd720e4b47599738c8162f951a5e. Latest consensus state: current_term: 1 leader_uuid: "abf4cd720e4b47599738c8162f951a5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abf4cd720e4b47599738c8162f951a5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34403 } } }
I20260812 09:49:31.294474 15753 sys_catalog.cc:458] T 00000000000000000000000000000000 P abf4cd720e4b47599738c8162f951a5e [sys.catalog]: This master's current role is: LEADER
I20260812 09:49:31.294868 15752 sys_catalog.cc:455] T 00000000000000000000000000000000 P abf4cd720e4b47599738c8162f951a5e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "abf4cd720e4b47599738c8162f951a5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abf4cd720e4b47599738c8162f951a5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34403 } } }
I20260812 09:49:31.294960 15752 sys_catalog.cc:458] T 00000000000000000000000000000000 P abf4cd720e4b47599738c8162f951a5e [sys.catalog]: This master's current role is: LEADER
I20260812 09:49:31.295413 15762 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 09:49:31.298753 15762 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 09:49:31.304723 15744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:31.295283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53353 (local address 127.0.0.1:34403)
0812 09:49:31.295383 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:31.295387 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:31.295905 (+ 518us) server_negotiation.cc:407] Connection header received
0812 09:49:31.296023 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:31.296028 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:31.296076 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:31.296145 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:49:31.297113 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:31.297853 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:31.299006 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:31.299322 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:31.299398 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:31.301224 (+ 1826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:49:31.301228 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:49:31.301230 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:49:31.301262 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:49:31.302709 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:49:31.303204 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:49:31.303210 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:49:31.303211 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:49:31.303314 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:49:31.303750 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:49:31.303753 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:49:31.303755 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:49:31.303928 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:49:31.304105 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:31.304428 (+ 323us) server_negotiation.cc:299] Negotiation successful
0812 09:49:31.304572 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260812 09:49:31.305577 15762 catalog_manager.cc:1347] Generated new cluster ID: 82c861092bbe435f9cdfd7e88cead094
I20260812 09:49:31.305635 15762 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 09:49:31.312789 15762 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 09:49:31.313905 15762 catalog_manager.cc:1504] Loading token signing keys...
I20260812 09:49:31.318781 15762 catalog_manager.cc:5868] T 00000000000000000000000000000000 P abf4cd720e4b47599738c8162f951a5e: Generated new TSK 0
I20260812 09:49:31.319502 15762 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 09:49:31.326004 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 15638
W20260812 09:49:31.341341 15604 connection.cc:535] client connection to 127.0.0.1:34403 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34403): Connection reset by peer (error 104)
2026-08-12T09:49:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1663 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786527905902985-13708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 09:49:31 dist-test-slave-jq4t krb5kdc[15804](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 09:49:31 dist-test-slave-jq4t krb5kdc[15804](info): set up 2 sockets
Aug 12 09:49:31 dist-test-slave-jq4t krb5kdc[15804](info): commencing operation
krb5kdc: starting...
W20260812 09:49:32.616354 13708 mini_kdc.cc:121] Time spent starting KDC: real 1.260s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786527905902985-13708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786527905902985-13708-0/krb5kdc/test-user.keytab.
Aug 12 09:49:32 dist-test-slave-jq4t krb5kdc[15804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528172, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T09:49:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T09:49:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786527905902985-13708-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786527905902985-13708-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 09:49:32.749342 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786527905902985-13708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786527905902985-13708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786527905902985-13708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44341
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786527905902985-13708-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44341
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:49:32.864804 15828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:49:32.865067 15828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:49:32.865108 15828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:49:32.869110 15828 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 09:49:32.869164 15828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:49:32.869184 15828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:49:32.869200 15828 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 09:49:32.869215 15828 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 09:49:32.874504 15828 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786527905902985-13708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786527905902985-13708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44341
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786527905902985-13708-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15828
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786527905902985-13708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:49:32.875510 15828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:49:32.876567 15828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:49:32.884812 15838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:49:32.884930 15828 server_base.cc:1034] running on GCE node
W20260812 09:49:32.884863 15837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:49:32.885120 15843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:49:32.885710 15828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:49:32.886758 15828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:49:32.887985 15828 hybrid_clock.cc:648] HybridClock initialized: now 1786528172887953 us; error 42 us; skew 500 ppm
Aug 12 09:49:32 dist-test-slave-jq4t krb5kdc[15804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528172, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:49:32.890846 15828 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 09:49:32.891673 15828 webserver.cc:460] Webserver started at http://127.0.0.1:35643/ using document root <none> and password file <none>
I20260812 09:49:32.892195 15828 fs_manager.cc:359] Metadata directory not provided
I20260812 09:49:32.892254 15828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:49:32.892418 15828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:49:32.894450 15828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786527905902985-13708-0/minicluster-data/master-0/data/instance:
uuid: "65c874e720554bea8fbe63a74cb02b8c"
format_stamp: "Formatted at 2026-08-12 09:49:32 on dist-test-slave-jq4t"
server_key: "f994f691fac4a8c294dc9d1e3d57a100"
server_key_iv: "5eb695b89d86f65679cb51a85322d776"
server_key_version: "encryptionkey@0"
I20260812 09:49:32.894974 15828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786527905902985-13708-0/minicluster-data/master-0/wal/instance:
uuid: "65c874e720554bea8fbe63a74cb02b8c"
format_stamp: "Formatted at 2026-08-12 09:49:32 on dist-test-slave-jq4t"
server_key: "f994f691fac4a8c294dc9d1e3d57a100"
server_key_iv: "5eb695b89d86f65679cb51a85322d776"
server_key_version: "encryptionkey@0"
I20260812 09:49:32.898638 15828 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260812 09:49:32.901078 15856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:49:32.902285 15828 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 09:49:32.902417 15828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786527905902985-13708-0/minicluster-data/master-0/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786527905902985-13708-0/minicluster-data/master-0/wal
uuid: "65c874e720554bea8fbe63a74cb02b8c"
format_stamp: "Formatted at 2026-08-12 09:49:32 on dist-test-slave-jq4t"
server_key: "f994f691fac4a8c294dc9d1e3d57a100"
server_key_iv: "5eb695b89d86f65679cb51a85322d776"
server_key_version: "encryptionkey@0"
I20260812 09:49:32.902519 15828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786527905902985-13708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786527905902985-13708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786527905902985-13708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:49:32.944096 15828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:49:32.947311 15828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:49:32.947528 15828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:49:32.954849 15828 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44341
I20260812 09:49:32.954875 15936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44341 every 8 connection(s)
I20260812 09:49:32.955847 15828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
I20260812 09:49:32.956247 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 15828
I20260812 09:49:32.956437 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786527905902985-13708-0/minicluster-data/master-0/wal/instance
I20260812 09:49:32.956779 13708 external_mini_cluster.cc:1428] Setting key d3bedcbbd0ee82e8bef6b734177d8b2a
I20260812 09:49:32.959364 15940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:49:32.966421 15940 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65c874e720554bea8fbe63a74cb02b8c: Bootstrap starting.
I20260812 09:49:32.967298 15949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:32.958250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37031 (local address 127.0.0.1:44341)
0812 09:49:32.958719 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:32.958739 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:32.958783 (+ 44us) server_negotiation.cc:407] Connection header received
0812 09:49:32.959652 (+ 869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:32.959675 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:32.960094 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:32.960496 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:49:32.960923 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:32.962490 (+ 1567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:32.963430 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:32.964106 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:32.964285 (+ 179us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:32.965288 (+ 1003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:49:32.965296 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:49:32.965315 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 09:49:32.965814 (+ 499us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:49:32.965886 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 09:49:32.965917 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:49:32.966056 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:32.966254 (+ 198us) server_negotiation.cc:299] Negotiation successful
0812 09:49:32.966399 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":140,"threads_started":1}
I20260812 09:49:32.969113 15940 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65c874e720554bea8fbe63a74cb02b8c: Neither blocks nor log segments found. Creating new log.
I20260812 09:49:32.970007 15940 log.cc:824] T 00000000000000000000000000000000 P 65c874e720554bea8fbe63a74cb02b8c: Log is configured to *not* fsync() on all Append() calls
I20260812 09:49:32.970381 13708 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 09:49:32.972018 15940 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65c874e720554bea8fbe63a74cb02b8c: No bootstrap required, opened a new log
I20260812 09:49:32.976912 15949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:32.972432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52559 (local address 127.0.0.1:44341)
0812 09:49:32.972563 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:32.972567 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:32.972940 (+ 373us) server_negotiation.cc:407] Connection header received
0812 09:49:32.973081 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:32.973083 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:32.973121 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:32.973191 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:49:32.973497 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:32.974010 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:32.975197 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:32.975401 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:32.975473 (+ 72us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:32.976320 (+ 847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:49:32.976323 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:49:32.976325 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 09:49:32.976339 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:49:32.976373 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 09:49:32.976384 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:49:32.976462 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:32.976700 (+ 238us) server_negotiation.cc:299] Negotiation successful
0812 09:49:32.976749 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260812 09:49:32.976861 15940 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65c874e720554bea8fbe63a74cb02b8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65c874e720554bea8fbe63a74cb02b8c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44341 } }
I20260812 09:49:32.977113 15940 raft_consensus.cc:374] T 00000000000000000000000000000000 P 65c874e720554bea8fbe63a74cb02b8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:49:32.977190 15940 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65c874e720554bea8fbe63a74cb02b8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65c874e720554bea8fbe63a74cb02b8c, State: Initialized, Role: FOLLOWER
I20260812 09:49:32.977615 15940 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65c874e720554bea8fbe63a74cb02b8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65c874e720554bea8fbe63a74cb02b8c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44341 } }
I20260812 09:49:32.977772 15940 raft_consensus.cc:388] T 00000000000000000000000000000000 P 65c874e720554bea8fbe63a74cb02b8c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 09:49:32.977869 15940 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65c874e720554bea8fbe63a74cb02b8c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 09:49:32.977986 15940 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65c874e720554bea8fbe63a74cb02b8c [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:49:32.978879 15940 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65c874e720554bea8fbe63a74cb02b8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65c874e720554bea8fbe63a74cb02b8c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44341 } }
I20260812 09:49:32.979213 15940 leader_election.cc:302] T 00000000000000000000000000000000 P 65c874e720554bea8fbe63a74cb02b8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 65c874e720554bea8fbe63a74cb02b8c; no voters:
I20260812 09:49:32.979509 15940 leader_election.cc:288] T 00000000000000000000000000000000 P 65c874e720554bea8fbe63a74cb02b8c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 09:49:32.979650 15960 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 65c874e720554bea8fbe63a74cb02b8c [term 1 FOLLOWER]: Leader election won for term 1
I20260812 09:49:32.979887 15960 raft_consensus.cc:686] T 00000000000000000000000000000000 P 65c874e720554bea8fbe63a74cb02b8c [term 1 LEADER]: Becoming Leader. State: Replica: 65c874e720554bea8fbe63a74cb02b8c, State: Running, Role: LEADER
I20260812 09:49:32.980208 15960 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65c874e720554bea8fbe63a74cb02b8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65c874e720554bea8fbe63a74cb02b8c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44341 } }
I20260812 09:49:32.980778 15940 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65c874e720554bea8fbe63a74cb02b8c [sys.catalog]: configured and running, proceeding with master startup.
I20260812 09:49:32.981977 15961 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65c874e720554bea8fbe63a74cb02b8c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "65c874e720554bea8fbe63a74cb02b8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65c874e720554bea8fbe63a74cb02b8c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44341 } } }
I20260812 09:49:32.982123 15961 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65c874e720554bea8fbe63a74cb02b8c [sys.catalog]: This master's current role is: LEADER
I20260812 09:49:32.982432 15964 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65c874e720554bea8fbe63a74cb02b8c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65c874e720554bea8fbe63a74cb02b8c. Latest consensus state: current_term: 1 leader_uuid: "65c874e720554bea8fbe63a74cb02b8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65c874e720554bea8fbe63a74cb02b8c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44341 } } }
I20260812 09:49:32.982548 15964 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65c874e720554bea8fbe63a74cb02b8c [sys.catalog]: This master's current role is: LEADER
I20260812 09:49:32.982636 15977 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 09:49:32.985680 15977 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 09:49:32.991086 15977 catalog_manager.cc:1347] Generated new cluster ID: 96f6959d0e7a4c1ebb4337af02b3d191
I20260812 09:49:32.991159 15977 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 09:49:33.007882 15977 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 09:49:33.008942 15977 catalog_manager.cc:1504] Loading token signing keys...
I20260812 09:49:33.013074 15977 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 65c874e720554bea8fbe63a74cb02b8c: Generated new TSK 0
I20260812 09:49:33.013761 15977 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 09:49:33.022171 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 15828
W20260812 09:49:33.036989 15802 connection.cc:535] client connection to 127.0.0.1:44341 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44341): Connection reset by peer (error 104)
2026-08-12T09:49:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1695 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786527905902985-13708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 09:49:33 dist-test-slave-jq4t krb5kdc[16000](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 09:49:33 dist-test-slave-jq4t krb5kdc[16000](info): set up 2 sockets
Aug 12 09:49:33 dist-test-slave-jq4t krb5kdc[16000](info): commencing operation
krb5kdc: starting...
W20260812 09:49:34.291148 13708 mini_kdc.cc:121] Time spent starting KDC: real 1.240s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786527905902985-13708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786527905902985-13708-0/krb5kdc/test-user.keytab.
Aug 12 09:49:34 dist-test-slave-jq4t krb5kdc[16000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528174, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T09:49:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T09:49:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786527905902985-13708-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786527905902985-13708-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 09:49:34.423825 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786527905902985-13708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786527905902985-13708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786527905902985-13708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42719
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786527905902985-13708-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42719
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:49:34.537658 16017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:49:34.537930 16017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:49:34.537972 16017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:49:34.541870 16017 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 09:49:34.541922 16017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:49:34.541942 16017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:49:34.541958 16017 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 09:49:34.541973 16017 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 09:49:34.547170 16017 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786527905902985-13708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786527905902985-13708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42719
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786527905902985-13708-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42719
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16017
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786527905902985-13708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:49:34.548226 16017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:49:34.549264 16017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:49:34.557683 16028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:49:34.557706 16025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:49:34.557719 16024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:49:34.557972 16017 server_base.cc:1034] running on GCE node
I20260812 09:49:34.558586 16017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:49:34.559655 16017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:49:34.560818 16017 hybrid_clock.cc:648] HybridClock initialized: now 1786528174560788 us; error 41 us; skew 500 ppm
Aug 12 09:49:34 dist-test-slave-jq4t krb5kdc[16000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528174, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:49:34.567687 16017 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 09:49:34.568722 16017 webserver.cc:460] Webserver started at http://127.0.0.1:33335/ using document root <none> and password file <none>
I20260812 09:49:34.569294 16017 fs_manager.cc:359] Metadata directory not provided
I20260812 09:49:34.569388 16017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:49:34.569639 16017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:49:34.571745 16017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786527905902985-13708-0/minicluster-data/master-0/data/instance:
uuid: "8fecd2b2469b4d04b24311c373994f7d"
format_stamp: "Formatted at 2026-08-12 09:49:34 on dist-test-slave-jq4t"
server_key: "217872ba863937a15c1bb8c02cddb790"
server_key_iv: "194d6c7ab380655ab435bdcb82b79318"
server_key_version: "encryptionkey@0"
I20260812 09:49:34.572350 16017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786527905902985-13708-0/minicluster-data/master-0/wal/instance:
uuid: "8fecd2b2469b4d04b24311c373994f7d"
format_stamp: "Formatted at 2026-08-12 09:49:34 on dist-test-slave-jq4t"
server_key: "217872ba863937a15c1bb8c02cddb790"
server_key_iv: "194d6c7ab380655ab435bdcb82b79318"
server_key_version: "encryptionkey@0"
I20260812 09:49:34.576109 16017 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260812 09:49:34.578569 16045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:49:34.579770 16017 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 09:49:34.579916 16017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786527905902985-13708-0/minicluster-data/master-0/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786527905902985-13708-0/minicluster-data/master-0/wal
uuid: "8fecd2b2469b4d04b24311c373994f7d"
format_stamp: "Formatted at 2026-08-12 09:49:34 on dist-test-slave-jq4t"
server_key: "217872ba863937a15c1bb8c02cddb790"
server_key_iv: "194d6c7ab380655ab435bdcb82b79318"
server_key_version: "encryptionkey@0"
I20260812 09:49:34.580024 16017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786527905902985-13708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786527905902985-13708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786527905902985-13708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:49:34.584619 16017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:49:34.587811 16017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:49:34.588006 16017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:49:34.594928 16017 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42719
I20260812 09:49:34.594961 16133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42719 every 8 connection(s)
I20260812 09:49:34.595829 16017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
I20260812 09:49:34.598642 16135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:49:34.600786 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 16017
I20260812 09:49:34.600941 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786527905902985-13708-0/minicluster-data/master-0/wal/instance
I20260812 09:49:34.601233 13708 external_mini_cluster.cc:1428] Setting key 0b525890ac131d8b763192ea06f79dba
I20260812 09:49:34.605484 16135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fecd2b2469b4d04b24311c373994f7d: Bootstrap starting.
I20260812 09:49:34.608088 16140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:34.602810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41249 (local address 127.0.0.1:42719)
0812 09:49:34.603381 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:34.603418 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:34.603464 (+ 46us) server_negotiation.cc:407] Connection header received
0812 09:49:34.604465 (+ 1001us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:34.604523 (+ 58us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:34.604952 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:34.605343 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:49:34.606899 (+ 1556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:49:34.606904 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:49:34.606916 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 09:49:34.606928 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:49:34.606984 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 09:49:34.607004 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:49:34.607168 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:34.607241 (+ 73us) server_negotiation.cc:299] Negotiation successful
0812 09:49:34.607403 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":157,"threads_started":1}
I20260812 09:49:34.608175 16135 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8fecd2b2469b4d04b24311c373994f7d: Neither blocks nor log segments found. Creating new log.
I20260812 09:49:34.609004 16135 log.cc:824] T 00000000000000000000000000000000 P 8fecd2b2469b4d04b24311c373994f7d: Log is configured to *not* fsync() on all Append() calls
I20260812 09:49:34.610998 13708 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 09:49:34.611277 16135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fecd2b2469b4d04b24311c373994f7d: No bootstrap required, opened a new log
I20260812 09:49:34.614842 16140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:34.613156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37503 (local address 127.0.0.1:42719)
0812 09:49:34.613270 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:34.613274 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:34.613443 (+ 169us) server_negotiation.cc:407] Connection header received
0812 09:49:34.613603 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:34.613607 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:34.613644 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:34.613725 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:49:34.614449 (+ 724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:49:34.614453 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:49:34.614456 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 09:49:34.614468 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:49:34.614495 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 09:49:34.614514 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:49:34.614608 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:34.614682 (+ 74us) server_negotiation.cc:299] Negotiation successful
0812 09:49:34.614722 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260812 09:49:34.615367 16135 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8fecd2b2469b4d04b24311c373994f7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fecd2b2469b4d04b24311c373994f7d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42719 } }
I20260812 09:49:34.615593 16135 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8fecd2b2469b4d04b24311c373994f7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:49:34.615671 16135 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8fecd2b2469b4d04b24311c373994f7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fecd2b2469b4d04b24311c373994f7d, State: Initialized, Role: FOLLOWER
I20260812 09:49:34.616178 16135 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8fecd2b2469b4d04b24311c373994f7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fecd2b2469b4d04b24311c373994f7d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42719 } }
I20260812 09:49:34.616361 16135 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8fecd2b2469b4d04b24311c373994f7d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 09:49:34.616436 16135 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8fecd2b2469b4d04b24311c373994f7d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 09:49:34.616582 16135 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8fecd2b2469b4d04b24311c373994f7d [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:49:34.617537 16135 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8fecd2b2469b4d04b24311c373994f7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fecd2b2469b4d04b24311c373994f7d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42719 } }
I20260812 09:49:34.617888 16135 leader_election.cc:302] T 00000000000000000000000000000000 P 8fecd2b2469b4d04b24311c373994f7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8fecd2b2469b4d04b24311c373994f7d; no voters:
I20260812 09:49:34.618270 16135 leader_election.cc:288] T 00000000000000000000000000000000 P 8fecd2b2469b4d04b24311c373994f7d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 09:49:34.618415 16150 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8fecd2b2469b4d04b24311c373994f7d [term 1 FOLLOWER]: Leader election won for term 1
I20260812 09:49:34.618688 16150 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8fecd2b2469b4d04b24311c373994f7d [term 1 LEADER]: Becoming Leader. State: Replica: 8fecd2b2469b4d04b24311c373994f7d, State: Running, Role: LEADER
I20260812 09:49:34.619071 16150 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8fecd2b2469b4d04b24311c373994f7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fecd2b2469b4d04b24311c373994f7d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42719 } }
I20260812 09:49:34.619618 16135 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8fecd2b2469b4d04b24311c373994f7d [sys.catalog]: configured and running, proceeding with master startup.
I20260812 09:49:34.620870 16152 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fecd2b2469b4d04b24311c373994f7d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8fecd2b2469b4d04b24311c373994f7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fecd2b2469b4d04b24311c373994f7d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42719 } } }
I20260812 09:49:34.621004 16152 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fecd2b2469b4d04b24311c373994f7d [sys.catalog]: This master's current role is: LEADER
I20260812 09:49:34.621392 16163 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 09:49:34.621436 16154 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fecd2b2469b4d04b24311c373994f7d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8fecd2b2469b4d04b24311c373994f7d. Latest consensus state: current_term: 1 leader_uuid: "8fecd2b2469b4d04b24311c373994f7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fecd2b2469b4d04b24311c373994f7d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42719 } } }
I20260812 09:49:34.621639 16154 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fecd2b2469b4d04b24311c373994f7d [sys.catalog]: This master's current role is: LEADER
I20260812 09:49:34.624722 16163 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 09:49:34.630254 16163 catalog_manager.cc:1347] Generated new cluster ID: 23b9e7099ef24fb3a676eee5b6c5f26a
I20260812 09:49:34.630339 16163 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 09:49:34.651568 16163 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 09:49:34.651949 16163 catalog_manager.cc:1504] Loading token signing keys...
I20260812 09:49:34.658468 16163 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8fecd2b2469b4d04b24311c373994f7d: Generated new TSK 0
I20260812 09:49:34.659212 16163 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 09:49:34.663553 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 16017
2026-08-12T09:49:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1638 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786527905902985-13708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 09:49:34 dist-test-slave-jq4t krb5kdc[16185](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 09:49:34 dist-test-slave-jq4t krb5kdc[16185](info): set up 2 sockets
Aug 12 09:49:34 dist-test-slave-jq4t krb5kdc[16185](info): commencing operation
krb5kdc: starting...
W20260812 09:49:36.022897 13708 mini_kdc.cc:121] Time spent starting KDC: real 1.332s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786527905902985-13708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786527905902985-13708-0/krb5kdc/test-user.keytab.
Aug 12 09:49:36 dist-test-slave-jq4t krb5kdc[16185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528176, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T09:49:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T09:49:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786527905902985-13708-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786527905902985-13708-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 09:49:36.149492 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786527905902985-13708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786527905902985-13708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786527905902985-13708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43705
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786527905902985-13708-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43705
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:49:36.261902 16203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:49:36.262166 16203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:49:36.262207 16203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:49:36.266170 16203 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 09:49:36.266227 16203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:49:36.266245 16203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:49:36.266263 16203 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 09:49:36.266278 16203 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 09:49:36.271549 16203 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786527905902985-13708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786527905902985-13708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43705
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786527905902985-13708-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43705
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16203
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786527905902985-13708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:49:36.272657 16203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:49:36.273664 16203 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:49:36.282470 16212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:49:36.282473 16217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:49:36.282473 16211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:49:36.282646 16203 server_base.cc:1034] running on GCE node
I20260812 09:49:36.283392 16203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:49:36.284533 16203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:49:36.285738 16203 hybrid_clock.cc:648] HybridClock initialized: now 1786528176285709 us; error 52 us; skew 500 ppm
Aug 12 09:49:36 dist-test-slave-jq4t krb5kdc[16185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528176, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:49:36.288717 16203 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 09:49:36.289706 16203 webserver.cc:460] Webserver started at http://127.0.0.1:39645/ using document root <none> and password file <none>
I20260812 09:49:36.290300 16203 fs_manager.cc:359] Metadata directory not provided
I20260812 09:49:36.290398 16203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:49:36.290637 16203 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:49:36.292798 16203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786527905902985-13708-0/minicluster-data/master-0/data/instance:
uuid: "e3b24ab3967f40019f7cfc76f81a343f"
format_stamp: "Formatted at 2026-08-12 09:49:36 on dist-test-slave-jq4t"
server_key: "cc5faf2d25717670857e9a292bb15a54"
server_key_iv: "7c055dd2061795499cdbe948820c9b5b"
server_key_version: "encryptionkey@0"
I20260812 09:49:36.293382 16203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786527905902985-13708-0/minicluster-data/master-0/wal/instance:
uuid: "e3b24ab3967f40019f7cfc76f81a343f"
format_stamp: "Formatted at 2026-08-12 09:49:36 on dist-test-slave-jq4t"
server_key: "cc5faf2d25717670857e9a292bb15a54"
server_key_iv: "7c055dd2061795499cdbe948820c9b5b"
server_key_version: "encryptionkey@0"
I20260812 09:49:36.297101 16203 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20260812 09:49:36.299561 16229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:49:36.300848 16203 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.003s
I20260812 09:49:36.300995 16203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786527905902985-13708-0/minicluster-data/master-0/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786527905902985-13708-0/minicluster-data/master-0/wal
uuid: "e3b24ab3967f40019f7cfc76f81a343f"
format_stamp: "Formatted at 2026-08-12 09:49:36 on dist-test-slave-jq4t"
server_key: "cc5faf2d25717670857e9a292bb15a54"
server_key_iv: "7c055dd2061795499cdbe948820c9b5b"
server_key_version: "encryptionkey@0"
I20260812 09:49:36.301108 16203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786527905902985-13708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786527905902985-13708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786527905902985-13708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:49:36.332448 16203 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:49:36.335620 16203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:49:36.335846 16203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:49:36.343403 16203 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43705
I20260812 09:49:36.343434 16320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43705 every 8 connection(s)
I20260812 09:49:36.344400 16203 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
I20260812 09:49:36.345943 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 16203
I20260812 09:49:36.346078 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786527905902985-13708-0/minicluster-data/master-0/wal/instance
I20260812 09:49:36.346377 13708 external_mini_cluster.cc:1428] Setting key e67585070f5b5c5aaf54b003019b707e
I20260812 09:49:36.347661 16321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:49:36.354645 16321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3b24ab3967f40019f7cfc76f81a343f: Bootstrap starting.
Aug 12 09:49:36 dist-test-slave-jq4t krb5kdc[16185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528176, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260812 09:49:36.357399 16321 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e3b24ab3967f40019f7cfc76f81a343f: Neither blocks nor log segments found. Creating new log.
I20260812 09:49:36.358178 16321 log.cc:824] T 00000000000000000000000000000000 P e3b24ab3967f40019f7cfc76f81a343f: Log is configured to *not* fsync() on all Append() calls
I20260812 09:49:36.360273 16321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3b24ab3967f40019f7cfc76f81a343f: No bootstrap required, opened a new log
I20260812 09:49:36.362112 16326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:36.347889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60011 (local address 127.0.0.1:43705)
0812 09:49:36.348322 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:36.348340 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:36.348386 (+ 46us) server_negotiation.cc:407] Connection header received
0812 09:49:36.349096 (+ 710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:36.349113 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:36.349450 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:36.349731 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:49:36.350742 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:36.351809 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:36.353004 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:36.353472 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:36.353607 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:36.356472 (+ 2865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:49:36.356523 (+ 51us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:49:36.356556 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:49:36.356587 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:49:36.358845 (+ 2258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:49:36.359390 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:49:36.359396 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:49:36.359402 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:49:36.359525 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:49:36.359888 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:49:36.359898 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:49:36.359900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:49:36.360340 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:49:36.360617 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:36.360938 (+ 321us) server_negotiation.cc:299] Negotiation successful
0812 09:49:36.361277 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":149,"threads_started":1}
I20260812 09:49:36.364434 16321 raft_consensus.cc:348] T 00000000000000000000000000000000 P e3b24ab3967f40019f7cfc76f81a343f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3b24ab3967f40019f7cfc76f81a343f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43705 } }
I20260812 09:49:36.364777 16321 raft_consensus.cc:374] T 00000000000000000000000000000000 P e3b24ab3967f40019f7cfc76f81a343f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:49:36.364909 16321 raft_consensus.cc:729] T 00000000000000000000000000000000 P e3b24ab3967f40019f7cfc76f81a343f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3b24ab3967f40019f7cfc76f81a343f, State: Initialized, Role: FOLLOWER
I20260812 09:49:36.365445 13708 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 09:49:36.365432 16321 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3b24ab3967f40019f7cfc76f81a343f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3b24ab3967f40019f7cfc76f81a343f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43705 } }
I20260812 09:49:36.365595 16321 raft_consensus.cc:388] T 00000000000000000000000000000000 P e3b24ab3967f40019f7cfc76f81a343f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 09:49:36.365664 16321 raft_consensus.cc:482] T 00000000000000000000000000000000 P e3b24ab3967f40019f7cfc76f81a343f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 09:49:36.365768 16321 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e3b24ab3967f40019f7cfc76f81a343f [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:49:36.366767 16321 raft_consensus.cc:504] T 00000000000000000000000000000000 P e3b24ab3967f40019f7cfc76f81a343f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3b24ab3967f40019f7cfc76f81a343f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43705 } }
I20260812 09:49:36.367125 16321 leader_election.cc:302] T 00000000000000000000000000000000 P e3b24ab3967f40019f7cfc76f81a343f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e3b24ab3967f40019f7cfc76f81a343f; no voters:
I20260812 09:49:36.367465 16321 leader_election.cc:288] T 00000000000000000000000000000000 P e3b24ab3967f40019f7cfc76f81a343f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 09:49:36.367569 16334 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e3b24ab3967f40019f7cfc76f81a343f [term 1 FOLLOWER]: Leader election won for term 1
I20260812 09:49:36.367806 16334 raft_consensus.cc:686] T 00000000000000000000000000000000 P e3b24ab3967f40019f7cfc76f81a343f [term 1 LEADER]: Becoming Leader. State: Replica: e3b24ab3967f40019f7cfc76f81a343f, State: Running, Role: LEADER
I20260812 09:49:36.368175 16334 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3b24ab3967f40019f7cfc76f81a343f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3b24ab3967f40019f7cfc76f81a343f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43705 } }
I20260812 09:49:36.369050 16321 sys_catalog.cc:564] T 00000000000000000000000000000000 P e3b24ab3967f40019f7cfc76f81a343f [sys.catalog]: configured and running, proceeding with master startup.
I20260812 09:49:36.370059 16340 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3b24ab3967f40019f7cfc76f81a343f [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3b24ab3967f40019f7cfc76f81a343f. Latest consensus state: current_term: 1 leader_uuid: "e3b24ab3967f40019f7cfc76f81a343f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3b24ab3967f40019f7cfc76f81a343f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43705 } } }
I20260812 09:49:36.370182 16340 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3b24ab3967f40019f7cfc76f81a343f [sys.catalog]: This master's current role is: LEADER
I20260812 09:49:36.370553 16335 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3b24ab3967f40019f7cfc76f81a343f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e3b24ab3967f40019f7cfc76f81a343f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3b24ab3967f40019f7cfc76f81a343f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43705 } } }
I20260812 09:49:36.370712 16335 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3b24ab3967f40019f7cfc76f81a343f [sys.catalog]: This master's current role is: LEADER
I20260812 09:49:36.371063 16353 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 09:49:36.375211 16353 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 09:49:36.377802 16326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:36.367698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36125 (local address 127.0.0.1:43705)
0812 09:49:36.367839 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:36.367844 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:36.368511 (+ 667us) server_negotiation.cc:407] Connection header received
0812 09:49:36.368680 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:36.368683 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:36.368734 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:36.368817 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:49:36.369739 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:36.370624 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:36.371872 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:36.372205 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:36.372318 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:36.374024 (+ 1706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:49:36.374028 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:49:36.374031 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:49:36.374064 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:49:36.375832 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:49:36.376335 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:49:36.376339 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:49:36.376341 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:49:36.376438 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:49:36.376800 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:49:36.376804 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:49:36.376806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:49:36.377045 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:49:36.377177 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:36.377507 (+ 330us) server_negotiation.cc:299] Negotiation successful
0812 09:49:36.377644 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260812 09:49:36.381419 16353 catalog_manager.cc:1347] Generated new cluster ID: cd9be02360a346deb1611d3355603599
I20260812 09:49:36.381500 16353 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 09:49:36.399307 16353 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 09:49:36.400225 16353 catalog_manager.cc:1504] Loading token signing keys...
I20260812 09:49:36.413048 16353 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e3b24ab3967f40019f7cfc76f81a343f: Generated new TSK 0
I20260812 09:49:36.413775 16353 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 09:49:36.427672 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 16203
2026-08-12T09:49:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1768 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786527905902985-13708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 09:49:36 dist-test-slave-jq4t krb5kdc[16369](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 09:49:36 dist-test-slave-jq4t krb5kdc[16369](info): set up 2 sockets
Aug 12 09:49:36 dist-test-slave-jq4t krb5kdc[16369](info): commencing operation
krb5kdc: starting...
W20260812 09:49:37.720806 13708 mini_kdc.cc:121] Time spent starting KDC: real 1.262s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786527905902985-13708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786527905902985-13708-0/krb5kdc/test-user.keytab.
Aug 12 09:49:37 dist-test-slave-jq4t krb5kdc[16369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T09:49:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T09:49:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786527905902985-13708-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786527905902985-13708-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 09:49:37.853847 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786527905902985-13708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786527905902985-13708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786527905902985-13708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43631
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786527905902985-13708-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43631
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:49:37.968292 16399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:49:37.968582 16399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:49:37.968626 16399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:49:37.972625 16399 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 09:49:37.972685 16399 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:49:37.972705 16399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:49:37.972721 16399 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 09:49:37.972738 16399 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 09:49:37.977952 16399 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786527905902985-13708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786527905902985-13708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43631
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786527905902985-13708-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43631
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16399
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786527905902985-13708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:49:37.978945 16399 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:49:37.979946 16399 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:49:37.988727 16406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:49:37.988727 16407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:49:37.988816 16399 server_base.cc:1034] running on GCE node
W20260812 09:49:37.988742 16409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:49:37.989595 16399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:49:37.990736 16399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:49:37.991918 16399 hybrid_clock.cc:648] HybridClock initialized: now 1786528177991886 us; error 43 us; skew 500 ppm
Aug 12 09:49:37 dist-test-slave-jq4t krb5kdc[16369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528177, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:49:37.994866 16399 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 09:49:37.995822 16399 webserver.cc:460] Webserver started at http://127.0.0.1:41879/ using document root <none> and password file <none>
I20260812 09:49:37.996408 16399 fs_manager.cc:359] Metadata directory not provided
I20260812 09:49:37.996531 16399 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:49:37.996786 16399 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:49:37.999058 16399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786527905902985-13708-0/minicluster-data/master-0/data/instance:
uuid: "d4fd0f4435094d0baf812e4551822954"
format_stamp: "Formatted at 2026-08-12 09:49:37 on dist-test-slave-jq4t"
server_key: "bd73ab8b2ae23f05326fbdf9bf412fc3"
server_key_iv: "8cf3925a5b99a40c09f0d2c54438842f"
server_key_version: "encryptionkey@0"
I20260812 09:49:37.999670 16399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786527905902985-13708-0/minicluster-data/master-0/wal/instance:
uuid: "d4fd0f4435094d0baf812e4551822954"
format_stamp: "Formatted at 2026-08-12 09:49:37 on dist-test-slave-jq4t"
server_key: "bd73ab8b2ae23f05326fbdf9bf412fc3"
server_key_iv: "8cf3925a5b99a40c09f0d2c54438842f"
server_key_version: "encryptionkey@0"
I20260812 09:49:38.003559 16399 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260812 09:49:38.006234 16423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:49:38.007547 16399 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260812 09:49:38.007696 16399 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786527905902985-13708-0/minicluster-data/master-0/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786527905902985-13708-0/minicluster-data/master-0/wal
uuid: "d4fd0f4435094d0baf812e4551822954"
format_stamp: "Formatted at 2026-08-12 09:49:37 on dist-test-slave-jq4t"
server_key: "bd73ab8b2ae23f05326fbdf9bf412fc3"
server_key_iv: "8cf3925a5b99a40c09f0d2c54438842f"
server_key_version: "encryptionkey@0"
I20260812 09:49:38.007812 16399 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786527905902985-13708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786527905902985-13708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786527905902985-13708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:49:38.019241 16399 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:49:38.022287 16399 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:49:38.022495 16399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:49:38.029970 16399 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43631
I20260812 09:49:38.030000 16511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43631 every 8 connection(s)
I20260812 09:49:38.030956 16399 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
I20260812 09:49:38.033828 16512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:49:38.039625 16512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4fd0f4435094d0baf812e4551822954: Bootstrap starting.
I20260812 09:49:38.040792 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 16399
I20260812 09:49:38.040959 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786527905902985-13708-0/minicluster-data/master-0/wal/instance
I20260812 09:49:38.041251 13708 external_mini_cluster.cc:1428] Setting key 975981a100c8152f184597d3956b05e9
I20260812 09:49:38.042629 16512 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d4fd0f4435094d0baf812e4551822954: Neither blocks nor log segments found. Creating new log.
I20260812 09:49:38.043427 16512 log.cc:824] T 00000000000000000000000000000000 P d4fd0f4435094d0baf812e4551822954: Log is configured to *not* fsync() on all Append() calls
I20260812 09:49:38.045475 16512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4fd0f4435094d0baf812e4551822954: No bootstrap required, opened a new log
I20260812 09:49:38.048938 16512 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4fd0f4435094d0baf812e4551822954 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4fd0f4435094d0baf812e4551822954" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43631 } }
I20260812 09:49:38.049193 16512 raft_consensus.cc:374] T 00000000000000000000000000000000 P d4fd0f4435094d0baf812e4551822954 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:49:38.049263 16512 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4fd0f4435094d0baf812e4551822954 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4fd0f4435094d0baf812e4551822954, State: Initialized, Role: FOLLOWER
I20260812 09:49:38.049772 16512 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4fd0f4435094d0baf812e4551822954 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4fd0f4435094d0baf812e4551822954" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43631 } }
I20260812 09:49:38.049943 16512 raft_consensus.cc:388] T 00000000000000000000000000000000 P d4fd0f4435094d0baf812e4551822954 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 09:49:38.050034 16512 raft_consensus.cc:482] T 00000000000000000000000000000000 P d4fd0f4435094d0baf812e4551822954 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 09:49:38.050158 16512 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4fd0f4435094d0baf812e4551822954 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:49:38.050482 16517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:38.042455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44493 (local address 127.0.0.1:43631)
0812 09:49:38.042954 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:38.042969 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:38.043003 (+ 34us) server_negotiation.cc:407] Connection header received
0812 09:49:38.043719 (+ 716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:38.043738 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:38.044201 (+ 463us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:38.044589 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:49:38.045017 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:38.046068 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:38.046979 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:38.047462 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:38.047640 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:38.048542 (+ 902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:49:38.048554 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:49:38.048566 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 09:49:38.049005 (+ 439us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:49:38.049067 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 09:49:38.049091 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:49:38.049208 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:38.049323 (+ 115us) server_negotiation.cc:299] Negotiation successful
0812 09:49:38.049562 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":140,"threads_started":1}
I20260812 09:49:38.051355 16512 raft_consensus.cc:504] T 00000000000000000000000000000000 P d4fd0f4435094d0baf812e4551822954 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4fd0f4435094d0baf812e4551822954" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43631 } }
I20260812 09:49:38.051721 16512 leader_election.cc:302] T 00000000000000000000000000000000 P d4fd0f4435094d0baf812e4551822954 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d4fd0f4435094d0baf812e4551822954; no voters:
I20260812 09:49:38.052080 16512 leader_election.cc:288] T 00000000000000000000000000000000 P d4fd0f4435094d0baf812e4551822954 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 09:49:38.052188 16519 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d4fd0f4435094d0baf812e4551822954 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 09:49:38.052454 16519 raft_consensus.cc:686] T 00000000000000000000000000000000 P d4fd0f4435094d0baf812e4551822954 [term 1 LEADER]: Becoming Leader. State: Replica: d4fd0f4435094d0baf812e4551822954, State: Running, Role: LEADER
I20260812 09:49:38.052838 16519 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4fd0f4435094d0baf812e4551822954 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4fd0f4435094d0baf812e4551822954" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43631 } }
I20260812 09:49:38.053382 16512 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4fd0f4435094d0baf812e4551822954 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 09:49:38.053745 13708 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 09:49:38.054564 16520 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4fd0f4435094d0baf812e4551822954 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d4fd0f4435094d0baf812e4551822954" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4fd0f4435094d0baf812e4551822954" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43631 } } }
I20260812 09:49:38.054699 16520 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4fd0f4435094d0baf812e4551822954 [sys.catalog]: This master's current role is: LEADER
I20260812 09:49:38.055080 16521 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4fd0f4435094d0baf812e4551822954 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4fd0f4435094d0baf812e4551822954. Latest consensus state: current_term: 1 leader_uuid: "d4fd0f4435094d0baf812e4551822954" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4fd0f4435094d0baf812e4551822954" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43631 } } }
I20260812 09:49:38.055245 16521 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4fd0f4435094d0baf812e4551822954 [sys.catalog]: This master's current role is: LEADER
W20260812 09:49:38.058781 16549 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d4fd0f4435094d0baf812e4551822954: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 09:49:38.058867 16549 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 09:49:38.058926 16532 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 09:49:38.059782 16532 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 09:49:38.060447 16517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:38.056536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41081 (local address 127.0.0.1:43631)
0812 09:49:38.056645 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:38.056649 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:38.056790 (+ 141us) server_negotiation.cc:407] Connection header received
0812 09:49:38.056936 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:38.056939 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:38.056985 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:38.057073 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:49:38.057342 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:38.057905 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:38.058737 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:38.059087 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:38.059205 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:38.060004 (+ 799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:49:38.060008 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:49:38.060010 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 09:49:38.060022 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:49:38.060063 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 09:49:38.060074 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:49:38.060156 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:38.060249 (+ 93us) server_negotiation.cc:299] Negotiation successful
0812 09:49:38.060295 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260812 09:49:38.065284 16532 catalog_manager.cc:1347] Generated new cluster ID: 953fa73871b645ac8046223ce866c0f7
I20260812 09:49:38.065352 16532 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 09:49:38.074797 16532 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 09:49:38.075829 16532 catalog_manager.cc:1504] Loading token signing keys...
I20260812 09:49:38.086678 16532 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d4fd0f4435094d0baf812e4551822954: Generated new TSK 0
I20260812 09:49:38.087365 16532 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 09:49:38.094363 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 16399
2026-08-12T09:49:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1663 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786527905902985-13708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 09:49:38 dist-test-slave-jq4t krb5kdc[16563](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 09:49:38 dist-test-slave-jq4t krb5kdc[16563](info): set up 2 sockets
Aug 12 09:49:38 dist-test-slave-jq4t krb5kdc[16563](info): commencing operation
krb5kdc: starting...
W20260812 09:49:39.365990 13708 mini_kdc.cc:121] Time spent starting KDC: real 1.243s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786527905902985-13708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786527905902985-13708-0/krb5kdc/test-user.keytab.
Aug 12 09:49:39 dist-test-slave-jq4t krb5kdc[16563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T09:49:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T09:49:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786527905902985-13708-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786527905902985-13708-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 09:49:39.495034 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786527905902985-13708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786527905902985-13708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786527905902985-13708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45349
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786527905902985-13708-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45349
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:49:39.610770 16581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:49:39.611040 16581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:49:39.611080 16581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:49:39.615094 16581 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 09:49:39.615155 16581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:49:39.615175 16581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:49:39.615192 16581 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 09:49:39.615207 16581 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 09:49:39.621042 16581 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786527905902985-13708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786527905902985-13708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45349
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786527905902985-13708-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16581
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786527905902985-13708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:49:39.622248 16581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:49:39.623461 16581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:49:39.632453 16592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:49:39.632453 16590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:49:39.632524 16589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:49:39.632877 16581 server_base.cc:1034] running on GCE node
I20260812 09:49:39.633553 16581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:49:39.634636 16581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:49:39.635840 16581 hybrid_clock.cc:648] HybridClock initialized: now 1786528179635807 us; error 45 us; skew 500 ppm
Aug 12 09:49:39 dist-test-slave-jq4t krb5kdc[16563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528179, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:49:39.642665 16581 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 09:49:39.643635 16581 webserver.cc:460] Webserver started at http://127.0.0.1:41717/ using document root <none> and password file <none>
I20260812 09:49:39.644212 16581 fs_manager.cc:359] Metadata directory not provided
I20260812 09:49:39.644308 16581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:49:39.644603 16581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:49:39.646641 16581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786527905902985-13708-0/minicluster-data/master-0/data/instance:
uuid: "acb6f47b3f484f3b81fac6890411cdf3"
format_stamp: "Formatted at 2026-08-12 09:49:39 on dist-test-slave-jq4t"
server_key: "01155d335bc92aa1139779e3f71455aa"
server_key_iv: "ecef502a1a5ff672004826e75f8b967a"
server_key_version: "encryptionkey@0"
I20260812 09:49:39.647218 16581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786527905902985-13708-0/minicluster-data/master-0/wal/instance:
uuid: "acb6f47b3f484f3b81fac6890411cdf3"
format_stamp: "Formatted at 2026-08-12 09:49:39 on dist-test-slave-jq4t"
server_key: "01155d335bc92aa1139779e3f71455aa"
server_key_iv: "ecef502a1a5ff672004826e75f8b967a"
server_key_version: "encryptionkey@0"
I20260812 09:49:39.650925 16581 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260812 09:49:39.653378 16606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:49:39.654716 16581 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260812 09:49:39.654861 16581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786527905902985-13708-0/minicluster-data/master-0/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786527905902985-13708-0/minicluster-data/master-0/wal
uuid: "acb6f47b3f484f3b81fac6890411cdf3"
format_stamp: "Formatted at 2026-08-12 09:49:39 on dist-test-slave-jq4t"
server_key: "01155d335bc92aa1139779e3f71455aa"
server_key_iv: "ecef502a1a5ff672004826e75f8b967a"
server_key_version: "encryptionkey@0"
I20260812 09:49:39.654969 16581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786527905902985-13708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786527905902985-13708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786527905902985-13708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:49:39.659597 16581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:49:39.662849 16581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:49:39.663066 16581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:49:39.670387 16581 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45349
I20260812 09:49:39.670439 16703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45349 every 8 connection(s)
I20260812 09:49:39.671319 16581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
I20260812 09:49:39.671657 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 16581
I20260812 09:49:39.671816 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786527905902985-13708-0/minicluster-data/master-0/wal/instance
I20260812 09:49:39.672137 13708 external_mini_cluster.cc:1428] Setting key 2b3f771971e3008b39bd53c9dd3e7f80
I20260812 09:49:39.674568 16704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:49:39.677314 16711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:39.673600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60265 (local address 127.0.0.1:45349)
0812 09:49:39.674060 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:39.674074 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:39.674109 (+ 35us) server_negotiation.cc:407] Connection header received
0812 09:49:39.674775 (+ 666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:39.674793 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:39.675071 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:39.675407 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:49:39.676161 (+ 754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:49:39.676165 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:49:39.676178 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 09:49:39.676188 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:49:39.676237 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 09:49:39.676256 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:49:39.676407 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:39.676480 (+ 73us) server_negotiation.cc:299] Negotiation successful
0812 09:49:39.676672 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":137,"threads_started":1}
I20260812 09:49:39.680094 13708 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 09:49:39.680961 16704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P acb6f47b3f484f3b81fac6890411cdf3: Bootstrap starting.
I20260812 09:49:39.683699 16704 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P acb6f47b3f484f3b81fac6890411cdf3: Neither blocks nor log segments found. Creating new log.
I20260812 09:49:39.683954 16711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:39.682314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42113 (local address 127.0.0.1:45349)
0812 09:49:39.682421 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:39.682425 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:39.682667 (+ 242us) server_negotiation.cc:407] Connection header received
0812 09:49:39.682779 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:39.682781 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:39.682809 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:39.682872 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:49:39.683538 (+ 666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:49:39.683541 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:49:39.683543 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 09:49:39.683553 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:49:39.683577 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 09:49:39.683587 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:49:39.683654 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:39.683816 (+ 162us) server_negotiation.cc:299] Negotiation successful
0812 09:49:39.683857 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260812 09:49:39.684675 16704 log.cc:824] T 00000000000000000000000000000000 P acb6f47b3f484f3b81fac6890411cdf3: Log is configured to *not* fsync() on all Append() calls
I20260812 09:49:39.686583 16704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P acb6f47b3f484f3b81fac6890411cdf3: No bootstrap required, opened a new log
I20260812 09:49:39.689990 16704 raft_consensus.cc:348] T 00000000000000000000000000000000 P acb6f47b3f484f3b81fac6890411cdf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acb6f47b3f484f3b81fac6890411cdf3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45349 } }
I20260812 09:49:39.690233 16704 raft_consensus.cc:374] T 00000000000000000000000000000000 P acb6f47b3f484f3b81fac6890411cdf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:49:39.690317 16704 raft_consensus.cc:729] T 00000000000000000000000000000000 P acb6f47b3f484f3b81fac6890411cdf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acb6f47b3f484f3b81fac6890411cdf3, State: Initialized, Role: FOLLOWER
I20260812 09:49:39.690840 16704 consensus_queue.cc:260] T 00000000000000000000000000000000 P acb6f47b3f484f3b81fac6890411cdf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acb6f47b3f484f3b81fac6890411cdf3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45349 } }
I20260812 09:49:39.691016 16704 raft_consensus.cc:388] T 00000000000000000000000000000000 P acb6f47b3f484f3b81fac6890411cdf3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 09:49:39.691089 16704 raft_consensus.cc:482] T 00000000000000000000000000000000 P acb6f47b3f484f3b81fac6890411cdf3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 09:49:39.691188 16704 raft_consensus.cc:3037] T 00000000000000000000000000000000 P acb6f47b3f484f3b81fac6890411cdf3 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:49:39.692029 16704 raft_consensus.cc:504] T 00000000000000000000000000000000 P acb6f47b3f484f3b81fac6890411cdf3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acb6f47b3f484f3b81fac6890411cdf3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45349 } }
I20260812 09:49:39.692365 16704 leader_election.cc:302] T 00000000000000000000000000000000 P acb6f47b3f484f3b81fac6890411cdf3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: acb6f47b3f484f3b81fac6890411cdf3; no voters:
I20260812 09:49:39.692685 16704 leader_election.cc:288] T 00000000000000000000000000000000 P acb6f47b3f484f3b81fac6890411cdf3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 09:49:39.692780 16722 raft_consensus.cc:2781] T 00000000000000000000000000000000 P acb6f47b3f484f3b81fac6890411cdf3 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 09:49:39.693051 16722 raft_consensus.cc:686] T 00000000000000000000000000000000 P acb6f47b3f484f3b81fac6890411cdf3 [term 1 LEADER]: Becoming Leader. State: Replica: acb6f47b3f484f3b81fac6890411cdf3, State: Running, Role: LEADER
I20260812 09:49:39.693418 16722 consensus_queue.cc:237] T 00000000000000000000000000000000 P acb6f47b3f484f3b81fac6890411cdf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acb6f47b3f484f3b81fac6890411cdf3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45349 } }
I20260812 09:49:39.693958 16704 sys_catalog.cc:564] T 00000000000000000000000000000000 P acb6f47b3f484f3b81fac6890411cdf3 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 09:49:39.695086 16723 sys_catalog.cc:455] T 00000000000000000000000000000000 P acb6f47b3f484f3b81fac6890411cdf3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "acb6f47b3f484f3b81fac6890411cdf3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acb6f47b3f484f3b81fac6890411cdf3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45349 } } }
I20260812 09:49:39.695225 16723 sys_catalog.cc:458] T 00000000000000000000000000000000 P acb6f47b3f484f3b81fac6890411cdf3 [sys.catalog]: This master's current role is: LEADER
I20260812 09:49:39.695663 16733 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 09:49:39.695631 16724 sys_catalog.cc:455] T 00000000000000000000000000000000 P acb6f47b3f484f3b81fac6890411cdf3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader acb6f47b3f484f3b81fac6890411cdf3. Latest consensus state: current_term: 1 leader_uuid: "acb6f47b3f484f3b81fac6890411cdf3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acb6f47b3f484f3b81fac6890411cdf3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45349 } } }
I20260812 09:49:39.696283 16724 sys_catalog.cc:458] T 00000000000000000000000000000000 P acb6f47b3f484f3b81fac6890411cdf3 [sys.catalog]: This master's current role is: LEADER
I20260812 09:49:39.698438 16733 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 09:49:39.703637 16733 catalog_manager.cc:1347] Generated new cluster ID: 021f68b0b9fb4c58a7142c2b81bcfc19
I20260812 09:49:39.703704 16733 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 09:49:39.736599 16733 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 09:49:39.737000 16733 catalog_manager.cc:1504] Loading token signing keys...
I20260812 09:49:39.745679 16733 catalog_manager.cc:5868] T 00000000000000000000000000000000 P acb6f47b3f484f3b81fac6890411cdf3: Generated new TSK 0
I20260812 09:49:39.746490 16733 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 09:49:39.755448 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 16581
2026-08-12T09:49:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1664 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786527905902985-13708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 09:49:39 dist-test-slave-jq4t krb5kdc[16756](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 09:49:39 dist-test-slave-jq4t krb5kdc[16756](info): set up 2 sockets
Aug 12 09:49:39 dist-test-slave-jq4t krb5kdc[16756](info): commencing operation
krb5kdc: starting...
W20260812 09:49:41.023136 13708 mini_kdc.cc:121] Time spent starting KDC: real 1.235s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786527905902985-13708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786527905902985-13708-0/krb5kdc/test-user.keytab.
Aug 12 09:49:41 dist-test-slave-jq4t krb5kdc[16756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528181, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T09:49:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T09:49:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786527905902985-13708-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786527905902985-13708-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 09:49:41.150661 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786527905902985-13708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786527905902985-13708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786527905902985-13708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46341
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786527905902985-13708-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46341
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:49:41.262410 16774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:49:41.262672 16774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:49:41.262713 16774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:49:41.266592 16774 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 09:49:41.266644 16774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:49:41.266664 16774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:49:41.266680 16774 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 09:49:41.266695 16774 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 09:49:41.271785 16774 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786527905902985-13708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786527905902985-13708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46341
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786527905902985-13708-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786527905902985-13708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:49:41.272779 16774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:49:41.273739 16774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:49:41.282526 16789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:49:41.282672 16774 server_base.cc:1034] running on GCE node
W20260812 09:49:41.282523 16786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:49:41.282518 16784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:49:41.283489 16774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:49:41.284643 16774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:49:41.285815 16774 hybrid_clock.cc:648] HybridClock initialized: now 1786528181285796 us; error 32 us; skew 500 ppm
Aug 12 09:49:41 dist-test-slave-jq4t krb5kdc[16756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528181, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:49:41.288647 16774 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 09:49:41.289458 16774 webserver.cc:460] Webserver started at http://127.0.0.1:34915/ using document root <none> and password file <none>
I20260812 09:49:41.289980 16774 fs_manager.cc:359] Metadata directory not provided
I20260812 09:49:41.290037 16774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:49:41.290202 16774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:49:41.292208 16774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786527905902985-13708-0/minicluster-data/master-0/data/instance:
uuid: "266934d350844a36a1c92937880beb9f"
format_stamp: "Formatted at 2026-08-12 09:49:41 on dist-test-slave-jq4t"
server_key: "831af5584b732de0b92834a3e9a0277b"
server_key_iv: "1e6a83fe3660ee1243d30196b9d5ddbf"
server_key_version: "encryptionkey@0"
I20260812 09:49:41.292809 16774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786527905902985-13708-0/minicluster-data/master-0/wal/instance:
uuid: "266934d350844a36a1c92937880beb9f"
format_stamp: "Formatted at 2026-08-12 09:49:41 on dist-test-slave-jq4t"
server_key: "831af5584b732de0b92834a3e9a0277b"
server_key_iv: "1e6a83fe3660ee1243d30196b9d5ddbf"
server_key_version: "encryptionkey@0"
I20260812 09:49:41.296384 16774 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260812 09:49:41.298795 16799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:49:41.300074 16774 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 09:49:41.300235 16774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786527905902985-13708-0/minicluster-data/master-0/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786527905902985-13708-0/minicluster-data/master-0/wal
uuid: "266934d350844a36a1c92937880beb9f"
format_stamp: "Formatted at 2026-08-12 09:49:41 on dist-test-slave-jq4t"
server_key: "831af5584b732de0b92834a3e9a0277b"
server_key_iv: "1e6a83fe3660ee1243d30196b9d5ddbf"
server_key_version: "encryptionkey@0"
I20260812 09:49:41.300347 16774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786527905902985-13708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786527905902985-13708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786527905902985-13708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:49:41.326124 16774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:49:41.329326 16774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:49:41.329533 16774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:49:41.336848 16774 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46341
I20260812 09:49:41.336894 16886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46341 every 8 connection(s)
I20260812 09:49:41.337822 16774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
I20260812 09:49:41.340675 16887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:49:41.346215 16887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 266934d350844a36a1c92937880beb9f: Bootstrap starting.
I20260812 09:49:41.347000 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 16774
I20260812 09:49:41.347149 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786527905902985-13708-0/minicluster-data/master-0/wal/instance
I20260812 09:49:41.347505 13708 external_mini_cluster.cc:1428] Setting key a930df72615907ca93021e89c38a0d51
I20260812 09:49:41.349201 16887 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 266934d350844a36a1c92937880beb9f: Neither blocks nor log segments found. Creating new log.
I20260812 09:49:41.349941 16887 log.cc:824] T 00000000000000000000000000000000 P 266934d350844a36a1c92937880beb9f: Log is configured to *not* fsync() on all Append() calls
I20260812 09:49:41.352213 16887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 266934d350844a36a1c92937880beb9f: No bootstrap required, opened a new log
Aug 12 09:49:41 dist-test-slave-jq4t krb5kdc[16756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528181, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260812 09:49:41.356925 16887 raft_consensus.cc:348] T 00000000000000000000000000000000 P 266934d350844a36a1c92937880beb9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "266934d350844a36a1c92937880beb9f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46341 } }
I20260812 09:49:41.357151 16887 raft_consensus.cc:374] T 00000000000000000000000000000000 P 266934d350844a36a1c92937880beb9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:49:41.357236 16887 raft_consensus.cc:729] T 00000000000000000000000000000000 P 266934d350844a36a1c92937880beb9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 266934d350844a36a1c92937880beb9f, State: Initialized, Role: FOLLOWER
I20260812 09:49:41.357689 16887 consensus_queue.cc:260] T 00000000000000000000000000000000 P 266934d350844a36a1c92937880beb9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "266934d350844a36a1c92937880beb9f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46341 } }
I20260812 09:49:41.357847 16887 raft_consensus.cc:388] T 00000000000000000000000000000000 P 266934d350844a36a1c92937880beb9f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 09:49:41.357921 16887 raft_consensus.cc:482] T 00000000000000000000000000000000 P 266934d350844a36a1c92937880beb9f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 09:49:41.358048 16887 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 266934d350844a36a1c92937880beb9f [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:49:41.358932 16887 raft_consensus.cc:504] T 00000000000000000000000000000000 P 266934d350844a36a1c92937880beb9f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "266934d350844a36a1c92937880beb9f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46341 } }
I20260812 09:49:41.359279 16887 leader_election.cc:302] T 00000000000000000000000000000000 P 266934d350844a36a1c92937880beb9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 266934d350844a36a1c92937880beb9f; no voters:
I20260812 09:49:41.359650 16887 leader_election.cc:288] T 00000000000000000000000000000000 P 266934d350844a36a1c92937880beb9f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 09:49:41.359712 16899 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 266934d350844a36a1c92937880beb9f [term 1 FOLLOWER]: Leader election won for term 1
I20260812 09:49:41.359946 16899 raft_consensus.cc:686] T 00000000000000000000000000000000 P 266934d350844a36a1c92937880beb9f [term 1 LEADER]: Becoming Leader. State: Replica: 266934d350844a36a1c92937880beb9f, State: Running, Role: LEADER
I20260812 09:49:41.360301 16899 consensus_queue.cc:237] T 00000000000000000000000000000000 P 266934d350844a36a1c92937880beb9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "266934d350844a36a1c92937880beb9f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46341 } }
I20260812 09:49:41.360992 16887 sys_catalog.cc:564] T 00000000000000000000000000000000 P 266934d350844a36a1c92937880beb9f [sys.catalog]: configured and running, proceeding with master startup.
I20260812 09:49:41.362053 16901 sys_catalog.cc:455] T 00000000000000000000000000000000 P 266934d350844a36a1c92937880beb9f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "266934d350844a36a1c92937880beb9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "266934d350844a36a1c92937880beb9f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46341 } } }
I20260812 09:49:41.362195 16901 sys_catalog.cc:458] T 00000000000000000000000000000000 P 266934d350844a36a1c92937880beb9f [sys.catalog]: This master's current role is: LEADER
I20260812 09:49:41.362601 16916 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 09:49:41.363794 16897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:41.348750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.48:53409 (local address 127.0.0.1:46341)
0812 09:49:41.349159 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:41.349173 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:41.349206 (+ 33us) server_negotiation.cc:407] Connection header received
0812 09:49:41.349881 (+ 675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:41.349898 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:41.350291 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:41.350669 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:49:41.351622 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:41.352735 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:41.353912 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:41.354356 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:41.354481 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:41.357539 (+ 3058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:49:41.357546 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:49:41.357564 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:49:41.357601 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:49:41.359393 (+ 1792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:49:41.360095 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:49:41.360100 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:49:41.360106 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:49:41.360244 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:49:41.360684 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:49:41.360688 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:49:41.360689 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:49:41.361107 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:49:41.361351 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:41.362645 (+ 1294us) server_negotiation.cc:299] Negotiation successful
0812 09:49:41.362875 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":125,"threads_started":1}
I20260812 09:49:41.363889 16902 sys_catalog.cc:455] T 00000000000000000000000000000000 P 266934d350844a36a1c92937880beb9f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 266934d350844a36a1c92937880beb9f. Latest consensus state: current_term: 1 leader_uuid: "266934d350844a36a1c92937880beb9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "266934d350844a36a1c92937880beb9f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46341 } } }
I20260812 09:49:41.364015 16902 sys_catalog.cc:458] T 00000000000000000000000000000000 P 266934d350844a36a1c92937880beb9f [sys.catalog]: This master's current role is: LEADER
I20260812 09:49:41.365630 16916 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 09:49:41.367239 13708 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 09:49:41.371840 16916 catalog_manager.cc:1347] Generated new cluster ID: d7de911bcbee48f084ce483ed981a8b8
I20260812 09:49:41.371901 16916 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 09:49:41.378712 16897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:41.369324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.48:56345 (local address 127.0.0.1:46341)
0812 09:49:41.369418 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:41.369422 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:41.369718 (+ 296us) server_negotiation.cc:407] Connection header received
0812 09:49:41.369888 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:41.369892 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:41.369942 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:41.370038 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:49:41.370781 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:41.371811 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:41.372946 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:41.373309 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:41.373427 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:41.374991 (+ 1564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:49:41.374995 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:49:41.374999 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:49:41.375025 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:49:41.376879 (+ 1854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:49:41.377368 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:49:41.377372 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:49:41.377374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:49:41.377466 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:49:41.377798 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:49:41.377802 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:49:41.377804 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:49:41.378036 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:49:41.378184 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:41.378444 (+ 260us) server_negotiation.cc:299] Negotiation successful
0812 09:49:41.378546 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260812 09:49:41.385146 16916 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 09:49:41.386526 16916 catalog_manager.cc:1504] Loading token signing keys...
I20260812 09:49:41.391919 16916 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 266934d350844a36a1c92937880beb9f: Generated new TSK 0
I20260812 09:49:41.392617 16916 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 09:49:41.404680 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 16774
W20260812 09:49:41.420749 16753 connection.cc:535] client connection to 127.0.0.1:46341 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46341): Connection reset by peer (error 104)
2026-08-12T09:49:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1647 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786527905902985-13708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 09:49:41 dist-test-slave-jq4t krb5kdc[16955](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 09:49:41 dist-test-slave-jq4t krb5kdc[16955](info): set up 2 sockets
Aug 12 09:49:41 dist-test-slave-jq4t krb5kdc[16955](info): commencing operation
krb5kdc: starting...
W20260812 09:49:42.689775 13708 mini_kdc.cc:121] Time spent starting KDC: real 1.254s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786527905902985-13708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786527905902985-13708-0/krb5kdc/test-user.keytab.
Aug 12 09:49:42 dist-test-slave-jq4t krb5kdc[16955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T09:49:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T09:49:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786527905902985-13708-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786527905902985-13708-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 09:49:42.819365 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786527905902985-13708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786527905902985-13708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786527905902985-13708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40317
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786527905902985-13708-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40317
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:49:42.931890 16979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:49:42.932219 16979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:49:42.932297 16979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:49:42.936411 16979 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 09:49:42.936514 16979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:49:42.936584 16979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:49:42.936623 16979 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 09:49:42.936654 16979 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 09:49:42.941756 16979 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786527905902985-13708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786527905902985-13708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40317
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786527905902985-13708-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16979
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786527905902985-13708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:49:42.942788 16979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:49:42.943804 16979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:49:42.952373 16988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:49:42.952351 16984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:49:42.952550 16979 server_base.cc:1034] running on GCE node
W20260812 09:49:42.952360 16985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:49:42.953261 16979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:49:42.954295 16979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:49:42.955490 16979 hybrid_clock.cc:648] HybridClock initialized: now 1786528182955462 us; error 39 us; skew 500 ppm
Aug 12 09:49:42 dist-test-slave-jq4t krb5kdc[16955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528182, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:49:42.958324 16979 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 09:49:42.959185 16979 webserver.cc:460] Webserver started at http://127.0.0.1:38569/ using document root <none> and password file <none>
I20260812 09:49:42.959738 16979 fs_manager.cc:359] Metadata directory not provided
I20260812 09:49:42.959831 16979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:49:42.960072 16979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:49:42.962121 16979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786527905902985-13708-0/minicluster-data/master-0/data/instance:
uuid: "2e32d3941e904a22a557e6cdfad434c0"
format_stamp: "Formatted at 2026-08-12 09:49:42 on dist-test-slave-jq4t"
server_key: "65edec4a490de1834ee3dd57141bb94d"
server_key_iv: "c09e00fddee7ea323101f509e32157cf"
server_key_version: "encryptionkey@0"
I20260812 09:49:42.962687 16979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786527905902985-13708-0/minicluster-data/master-0/wal/instance:
uuid: "2e32d3941e904a22a557e6cdfad434c0"
format_stamp: "Formatted at 2026-08-12 09:49:42 on dist-test-slave-jq4t"
server_key: "65edec4a490de1834ee3dd57141bb94d"
server_key_iv: "c09e00fddee7ea323101f509e32157cf"
server_key_version: "encryptionkey@0"
I20260812 09:49:42.966441 16979 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260812 09:49:42.968801 17001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:49:42.970010 16979 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 09:49:42.970178 16979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786527905902985-13708-0/minicluster-data/master-0/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786527905902985-13708-0/minicluster-data/master-0/wal
uuid: "2e32d3941e904a22a557e6cdfad434c0"
format_stamp: "Formatted at 2026-08-12 09:49:42 on dist-test-slave-jq4t"
server_key: "65edec4a490de1834ee3dd57141bb94d"
server_key_iv: "c09e00fddee7ea323101f509e32157cf"
server_key_version: "encryptionkey@0"
I20260812 09:49:42.970290 16979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786527905902985-13708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786527905902985-13708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786527905902985-13708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:49:42.982954 16979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:49:42.986099 16979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:49:42.986328 16979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:49:42.994208 16979 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40317
I20260812 09:49:42.994235 17080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40317 every 8 connection(s)
I20260812 09:49:42.995275 16979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
I20260812 09:49:42.995890 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 16979
I20260812 09:49:42.996053 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786527905902985-13708-0/minicluster-data/master-0/wal/instance
I20260812 09:49:42.996336 13708 external_mini_cluster.cc:1428] Setting key 4fc7c6606327cba964c9f77d3e319367
I20260812 09:49:42.998805 17081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:49:43.006032 17081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e32d3941e904a22a557e6cdfad434c0: Bootstrap starting.
I20260812 09:49:43.006917 17086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:42.997835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.48:34699 (local address 127.0.0.1:40317)
0812 09:49:42.998314 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:42.998334 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:42.998380 (+ 46us) server_negotiation.cc:407] Connection header received
0812 09:49:42.999250 (+ 870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:42.999274 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:42.999712 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:43.000100 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:49:43.000558 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:43.002075 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:43.003023 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:43.003706 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:43.003883 (+ 177us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:43.004895 (+ 1012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:49:43.004905 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:49:43.004921 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 09:49:43.005425 (+ 504us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:49:43.005497 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 09:49:43.005526 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:49:43.005665 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:43.005879 (+ 214us) server_negotiation.cc:299] Negotiation successful
0812 09:49:43.006013 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":122,"threads_started":1}
I20260812 09:49:43.008683 17081 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e32d3941e904a22a557e6cdfad434c0: Neither blocks nor log segments found. Creating new log.
I20260812 09:49:43.009595 17081 log.cc:824] T 00000000000000000000000000000000 P 2e32d3941e904a22a557e6cdfad434c0: Log is configured to *not* fsync() on all Append() calls
I20260812 09:49:43.009925 13708 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 09:49:43.011621 17081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e32d3941e904a22a557e6cdfad434c0: No bootstrap required, opened a new log
I20260812 09:49:43.015345 17081 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e32d3941e904a22a557e6cdfad434c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e32d3941e904a22a557e6cdfad434c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40317 } }
I20260812 09:49:43.015614 17081 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2e32d3941e904a22a557e6cdfad434c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:49:43.015717 17081 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e32d3941e904a22a557e6cdfad434c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e32d3941e904a22a557e6cdfad434c0, State: Initialized, Role: FOLLOWER
I20260812 09:49:43.016206 17081 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e32d3941e904a22a557e6cdfad434c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e32d3941e904a22a557e6cdfad434c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40317 } }
I20260812 09:49:43.016402 17081 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e32d3941e904a22a557e6cdfad434c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 09:49:43.016503 17081 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e32d3941e904a22a557e6cdfad434c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 09:49:43.016619 17081 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e32d3941e904a22a557e6cdfad434c0 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:49:43.017114 17086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:43.012107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.48:39621 (local address 127.0.0.1:40317)
0812 09:49:43.012907 (+ 800us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:43.012910 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:43.012920 (+ 10us) server_negotiation.cc:407] Connection header received
0812 09:49:43.012949 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:43.012951 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:43.012997 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:43.013058 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:49:43.013368 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:43.014244 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:43.015366 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:43.015597 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:43.015742 (+ 145us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:43.016611 (+ 869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:49:43.016615 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:49:43.016618 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 09:49:43.016631 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:49:43.016663 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 09:49:43.016692 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:49:43.016787 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:43.016948 (+ 161us) server_negotiation.cc:299] Negotiation successful
0812 09:49:43.016984 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":736}
I20260812 09:49:43.017758 17081 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e32d3941e904a22a557e6cdfad434c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e32d3941e904a22a557e6cdfad434c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40317 } }
I20260812 09:49:43.018146 17081 leader_election.cc:302] T 00000000000000000000000000000000 P 2e32d3941e904a22a557e6cdfad434c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e32d3941e904a22a557e6cdfad434c0; no voters:
I20260812 09:49:43.018530 17081 leader_election.cc:288] T 00000000000000000000000000000000 P 2e32d3941e904a22a557e6cdfad434c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 09:49:43.018644 17096 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e32d3941e904a22a557e6cdfad434c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 09:49:43.018877 17096 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e32d3941e904a22a557e6cdfad434c0 [term 1 LEADER]: Becoming Leader. State: Replica: 2e32d3941e904a22a557e6cdfad434c0, State: Running, Role: LEADER
I20260812 09:49:43.019170 17096 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e32d3941e904a22a557e6cdfad434c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e32d3941e904a22a557e6cdfad434c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40317 } }
I20260812 09:49:43.019724 17081 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e32d3941e904a22a557e6cdfad434c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 09:49:43.020795 17097 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e32d3941e904a22a557e6cdfad434c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2e32d3941e904a22a557e6cdfad434c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e32d3941e904a22a557e6cdfad434c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40317 } } }
I20260812 09:49:43.020941 17097 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e32d3941e904a22a557e6cdfad434c0 [sys.catalog]: This master's current role is: LEADER
I20260812 09:49:43.021229 17099 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e32d3941e904a22a557e6cdfad434c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e32d3941e904a22a557e6cdfad434c0. Latest consensus state: current_term: 1 leader_uuid: "2e32d3941e904a22a557e6cdfad434c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e32d3941e904a22a557e6cdfad434c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40317 } } }
I20260812 09:49:43.021344 17099 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e32d3941e904a22a557e6cdfad434c0 [sys.catalog]: This master's current role is: LEADER
W20260812 09:49:43.024336 17122 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2e32d3941e904a22a557e6cdfad434c0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 09:49:43.024396 17122 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 09:49:43.024498 17113 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 09:49:43.025192 17113 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 09:49:43.030146 17113 catalog_manager.cc:1347] Generated new cluster ID: 63317aabf3e44330be32f4a75b2f28c6
I20260812 09:49:43.030211 17113 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 09:49:43.041325 17113 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 09:49:43.042184 17113 catalog_manager.cc:1504] Loading token signing keys...
I20260812 09:49:43.047394 17113 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2e32d3941e904a22a557e6cdfad434c0: Generated new TSK 0
I20260812 09:49:43.047999 17113 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 09:49:43.054163 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 16979
W20260812 09:49:43.074543 16951 connection.cc:535] client connection to 127.0.0.1:40317 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40317): Connection reset by peer (error 104)
2026-08-12T09:49:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1653 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786527905902985-13708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 09:49:43 dist-test-slave-jq4t krb5kdc[17139](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 09:49:43 dist-test-slave-jq4t krb5kdc[17139](info): set up 2 sockets
Aug 12 09:49:43 dist-test-slave-jq4t krb5kdc[17139](info): commencing operation
krb5kdc: starting...
W20260812 09:49:44.331859 13708 mini_kdc.cc:121] Time spent starting KDC: real 1.243s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786527905902985-13708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786527905902985-13708-0/krb5kdc/test-user.keytab.
Aug 12 09:49:44 dist-test-slave-jq4t krb5kdc[17139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T09:49:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T09:49:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786527905902985-13708-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786527905902985-13708-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 09:49:44.459296 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786527905902985-13708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786527905902985-13708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786527905902985-13708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37285
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786527905902985-13708-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37285
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:49:44.571223 17165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:49:44.571521 17165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:49:44.571600 17165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:49:44.575555 17165 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 09:49:44.575644 17165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:49:44.575682 17165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:49:44.575740 17165 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 09:49:44.575778 17165 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 09:49:44.580855 17165 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786527905902985-13708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786527905902985-13708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37285
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786527905902985-13708-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17165
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786527905902985-13708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:49:44.581871 17165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:49:44.582883 17165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:49:44.591899 17180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:49:44.591993 17174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:49:44.591957 17171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:49:44.591984 17165 server_base.cc:1034] running on GCE node
I20260812 09:49:44.592882 17165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:49:44.594023 17165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:49:44.595228 17165 hybrid_clock.cc:648] HybridClock initialized: now 1786528184595203 us; error 44 us; skew 500 ppm
Aug 12 09:49:44 dist-test-slave-jq4t krb5kdc[17139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528184, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:49:44.598171 17165 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 09:49:44.599095 17165 webserver.cc:460] Webserver started at http://127.0.0.1:45409/ using document root <none> and password file <none>
I20260812 09:49:44.599682 17165 fs_manager.cc:359] Metadata directory not provided
I20260812 09:49:44.599779 17165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:49:44.600024 17165 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:49:44.602190 17165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786527905902985-13708-0/minicluster-data/master-0/data/instance:
uuid: "1d8d8835afed4d55ab8bbcfabc3dca7a"
format_stamp: "Formatted at 2026-08-12 09:49:44 on dist-test-slave-jq4t"
server_key: "d2e94d6dd06b523108ad8d4ff4e326bc"
server_key_iv: "c1df735cc508d27762db5ab884572b8c"
server_key_version: "encryptionkey@0"
I20260812 09:49:44.602792 17165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786527905902985-13708-0/minicluster-data/master-0/wal/instance:
uuid: "1d8d8835afed4d55ab8bbcfabc3dca7a"
format_stamp: "Formatted at 2026-08-12 09:49:44 on dist-test-slave-jq4t"
server_key: "d2e94d6dd06b523108ad8d4ff4e326bc"
server_key_iv: "c1df735cc508d27762db5ab884572b8c"
server_key_version: "encryptionkey@0"
I20260812 09:49:44.606793 17165 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 09:49:44.609440 17192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:49:44.610723 17165 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 09:49:44.610889 17165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786527905902985-13708-0/minicluster-data/master-0/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786527905902985-13708-0/minicluster-data/master-0/wal
uuid: "1d8d8835afed4d55ab8bbcfabc3dca7a"
format_stamp: "Formatted at 2026-08-12 09:49:44 on dist-test-slave-jq4t"
server_key: "d2e94d6dd06b523108ad8d4ff4e326bc"
server_key_iv: "c1df735cc508d27762db5ab884572b8c"
server_key_version: "encryptionkey@0"
I20260812 09:49:44.611012 17165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786527905902985-13708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786527905902985-13708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786527905902985-13708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:49:44.615796 17165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:49:44.619256 17165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:49:44.619462 17165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:49:44.626788 17165 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37285
I20260812 09:49:44.626820 17285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37285 every 8 connection(s)
I20260812 09:49:44.627733 17165 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
I20260812 09:49:44.630561 17286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:49:44.635623 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 17165
I20260812 09:49:44.635737 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786527905902985-13708-0/minicluster-data/master-0/wal/instance
I20260812 09:49:44.636034 13708 external_mini_cluster.cc:1428] Setting key f8c36747fa41781b2287a765dec90c96
I20260812 09:49:44.636188 17286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d8d8835afed4d55ab8bbcfabc3dca7a: Bootstrap starting.
I20260812 09:49:44.638999 17286 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d8d8835afed4d55ab8bbcfabc3dca7a: Neither blocks nor log segments found. Creating new log.
I20260812 09:49:44.639890 17286 log.cc:824] T 00000000000000000000000000000000 P 1d8d8835afed4d55ab8bbcfabc3dca7a: Log is configured to *not* fsync() on all Append() calls
I20260812 09:49:44.641819 17291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:44.637452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.48:38033 (local address 127.0.0.1:37285)
0812 09:49:44.637894 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:44.637912 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:44.637964 (+ 52us) server_negotiation.cc:407] Connection header received
0812 09:49:44.638749 (+ 785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:44.638772 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:44.639141 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:44.639510 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:49:44.640299 (+ 789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:49:44.640306 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:49:44.640323 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 09:49:44.640338 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:49:44.640394 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 09:49:44.640413 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:49:44.640569 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:44.640668 (+ 99us) server_negotiation.cc:299] Negotiation successful
0812 09:49:44.640984 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":119,"threads_started":1}
I20260812 09:49:44.642194 17286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d8d8835afed4d55ab8bbcfabc3dca7a: No bootstrap required, opened a new log
I20260812 09:49:44.645097 13708 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 09:49:44.647333 17286 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d8d8835afed4d55ab8bbcfabc3dca7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d8d8835afed4d55ab8bbcfabc3dca7a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37285 } }
I20260812 09:49:44.647646 17286 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d8d8835afed4d55ab8bbcfabc3dca7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:49:44.647723 17286 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d8d8835afed4d55ab8bbcfabc3dca7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d8d8835afed4d55ab8bbcfabc3dca7a, State: Initialized, Role: FOLLOWER
I20260812 09:49:44.648185 17286 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d8d8835afed4d55ab8bbcfabc3dca7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d8d8835afed4d55ab8bbcfabc3dca7a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37285 } }
I20260812 09:49:44.648337 17286 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d8d8835afed4d55ab8bbcfabc3dca7a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 09:49:44.648409 17286 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d8d8835afed4d55ab8bbcfabc3dca7a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 09:49:44.648541 17286 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d8d8835afed4d55ab8bbcfabc3dca7a [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:49:44.649492 17286 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d8d8835afed4d55ab8bbcfabc3dca7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d8d8835afed4d55ab8bbcfabc3dca7a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37285 } }
I20260812 09:49:44.649840 17286 leader_election.cc:302] T 00000000000000000000000000000000 P 1d8d8835afed4d55ab8bbcfabc3dca7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d8d8835afed4d55ab8bbcfabc3dca7a; no voters:
I20260812 09:49:44.650151 17286 leader_election.cc:288] T 00000000000000000000000000000000 P 1d8d8835afed4d55ab8bbcfabc3dca7a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 09:49:44.650295 17307 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d8d8835afed4d55ab8bbcfabc3dca7a [term 1 FOLLOWER]: Leader election won for term 1
I20260812 09:49:44.650540 17307 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d8d8835afed4d55ab8bbcfabc3dca7a [term 1 LEADER]: Becoming Leader. State: Replica: 1d8d8835afed4d55ab8bbcfabc3dca7a, State: Running, Role: LEADER
I20260812 09:49:44.650950 17307 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d8d8835afed4d55ab8bbcfabc3dca7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d8d8835afed4d55ab8bbcfabc3dca7a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37285 } }
I20260812 09:49:44.651599 17286 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d8d8835afed4d55ab8bbcfabc3dca7a [sys.catalog]: configured and running, proceeding with master startup.
I20260812 09:49:44.651793 17291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:44.647110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.48:42395 (local address 127.0.0.1:37285)
0812 09:49:44.647245 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:44.647249 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:44.647465 (+ 216us) server_negotiation.cc:407] Connection header received
0812 09:49:44.647593 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:44.647595 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:44.647624 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:44.647718 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:49:44.651215 (+ 3497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:49:44.651218 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:49:44.651221 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 09:49:44.651233 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:49:44.651262 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 09:49:44.651274 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:49:44.651349 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:44.651632 (+ 283us) server_negotiation.cc:299] Negotiation successful
0812 09:49:44.651676 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260812 09:49:44.652842 17309 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d8d8835afed4d55ab8bbcfabc3dca7a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d8d8835afed4d55ab8bbcfabc3dca7a. Latest consensus state: current_term: 1 leader_uuid: "1d8d8835afed4d55ab8bbcfabc3dca7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d8d8835afed4d55ab8bbcfabc3dca7a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37285 } } }
I20260812 09:49:44.652971 17309 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d8d8835afed4d55ab8bbcfabc3dca7a [sys.catalog]: This master's current role is: LEADER
I20260812 09:49:44.653285 17308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d8d8835afed4d55ab8bbcfabc3dca7a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d8d8835afed4d55ab8bbcfabc3dca7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d8d8835afed4d55ab8bbcfabc3dca7a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37285 } } }
I20260812 09:49:44.653379 17308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d8d8835afed4d55ab8bbcfabc3dca7a [sys.catalog]: This master's current role is: LEADER
I20260812 09:49:44.653504 17319 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 09:49:44.656718 17319 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 09:49:44.662155 17319 catalog_manager.cc:1347] Generated new cluster ID: d54eb33f19634d299bb8f1b6afe7ffea
I20260812 09:49:44.662217 17319 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 09:49:44.686594 17319 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 09:49:44.686975 17319 catalog_manager.cc:1504] Loading token signing keys...
I20260812 09:49:44.694355 17319 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d8d8835afed4d55ab8bbcfabc3dca7a: Generated new TSK 0
I20260812 09:49:44.695116 17319 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 09:49:44.708398 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 17165
2026-08-12T09:49:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1650 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786527905902985-13708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 09:49:44 dist-test-slave-jq4t krb5kdc[17341](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 09:49:44 dist-test-slave-jq4t krb5kdc[17341](info): set up 2 sockets
Aug 12 09:49:44 dist-test-slave-jq4t krb5kdc[17341](info): commencing operation
krb5kdc: starting...
W20260812 09:49:45.987892 13708 mini_kdc.cc:121] Time spent starting KDC: real 1.248s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786527905902985-13708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786527905902985-13708-0/krb5kdc/test-user.keytab.
Aug 12 09:49:46 dist-test-slave-jq4t krb5kdc[17341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528186, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T09:49:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T09:49:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786527905902985-13708-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786527905902985-13708-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 09:49:46.115675 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786527905902985-13708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786527905902985-13708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786527905902985-13708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33271
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786527905902985-13708-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33271
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:49:46.227044 17361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:49:46.227298 17361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:49:46.227339 17361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:49:46.231196 17361 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 09:49:46.231247 17361 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:49:46.231266 17361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:49:46.231282 17361 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 09:49:46.231297 17361 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 09:49:46.236303 17361 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786527905902985-13708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786527905902985-13708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33271
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786527905902985-13708-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17361
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786527905902985-13708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:49:46.237354 17361 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:49:46.238353 17361 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 09:49:46.247360 17361 server_base.cc:1034] running on GCE node
W20260812 09:49:46.247346 17370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:49:46.247309 17369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:49:46.247309 17373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:49:46.248131 17361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:49:46.249266 17361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:49:46.250483 17361 hybrid_clock.cc:648] HybridClock initialized: now 1786528186250463 us; error 31 us; skew 500 ppm
Aug 12 09:49:46 dist-test-slave-jq4t krb5kdc[17341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528186, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:49:46.253531 17361 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 09:49:46.254407 17361 webserver.cc:460] Webserver started at http://127.0.0.1:33091/ using document root <none> and password file <none>
I20260812 09:49:46.254974 17361 fs_manager.cc:359] Metadata directory not provided
I20260812 09:49:46.255069 17361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:49:46.255308 17361 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:49:46.257550 17361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786527905902985-13708-0/minicluster-data/master-0/data/instance:
uuid: "7387a72c0b444d6ea88a911c8b52ae3f"
format_stamp: "Formatted at 2026-08-12 09:49:46 on dist-test-slave-jq4t"
server_key: "0288987f7064ab0cfa006415c3efb9eb"
server_key_iv: "112f9e69e73ff8c25fd19f93f0f23e8b"
server_key_version: "encryptionkey@0"
I20260812 09:49:46.258127 17361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786527905902985-13708-0/minicluster-data/master-0/wal/instance:
uuid: "7387a72c0b444d6ea88a911c8b52ae3f"
format_stamp: "Formatted at 2026-08-12 09:49:46 on dist-test-slave-jq4t"
server_key: "0288987f7064ab0cfa006415c3efb9eb"
server_key_iv: "112f9e69e73ff8c25fd19f93f0f23e8b"
server_key_version: "encryptionkey@0"
I20260812 09:49:46.261852 17361 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 09:49:46.264189 17385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:49:46.265429 17361 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 09:49:46.265583 17361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786527905902985-13708-0/minicluster-data/master-0/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786527905902985-13708-0/minicluster-data/master-0/wal
uuid: "7387a72c0b444d6ea88a911c8b52ae3f"
format_stamp: "Formatted at 2026-08-12 09:49:46 on dist-test-slave-jq4t"
server_key: "0288987f7064ab0cfa006415c3efb9eb"
server_key_iv: "112f9e69e73ff8c25fd19f93f0f23e8b"
server_key_version: "encryptionkey@0"
I20260812 09:49:46.265695 17361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786527905902985-13708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786527905902985-13708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786527905902985-13708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:49:46.289201 17361 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:49:46.292280 17361 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:49:46.292518 17361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:49:46.299981 17361 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33271
I20260812 09:49:46.300001 17479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33271 every 8 connection(s)
I20260812 09:49:46.301004 17361 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
I20260812 09:49:46.302374 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 17361
I20260812 09:49:46.302532 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786527905902985-13708-0/minicluster-data/master-0/wal/instance
I20260812 09:49:46.302817 13708 external_mini_cluster.cc:1428] Setting key 28a2b2555a4e8126d02a4e3fe9c593c1
I20260812 09:49:46.304083 17480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:49:46.309872 17480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7387a72c0b444d6ea88a911c8b52ae3f: Bootstrap starting.
Aug 12 09:49:46 dist-test-slave-jq4t krb5kdc[17341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528186, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260812 09:49:46.312304 17480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7387a72c0b444d6ea88a911c8b52ae3f: Neither blocks nor log segments found. Creating new log.
I20260812 09:49:46.313246 17480 log.cc:824] T 00000000000000000000000000000000 P 7387a72c0b444d6ea88a911c8b52ae3f: Log is configured to *not* fsync() on all Append() calls
I20260812 09:49:46.315215 17480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7387a72c0b444d6ea88a911c8b52ae3f: No bootstrap required, opened a new log
I20260812 09:49:46.317240 17483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:46.304275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.48:43497 (local address 127.0.0.1:33271)
0812 09:49:46.304743 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:46.304761 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:46.304800 (+ 39us) server_negotiation.cc:407] Connection header received
0812 09:49:46.305436 (+ 636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:46.305451 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:46.305799 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:46.306159 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:49:46.306996 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:46.308005 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:46.308959 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:46.309443 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:46.309640 (+ 197us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:46.312367 (+ 2727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:49:46.312373 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:49:46.312386 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:49:46.312438 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:49:46.314258 (+ 1820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:49:46.314806 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:49:46.314813 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:49:46.314825 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:49:46.314936 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:49:46.315309 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:49:46.315312 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:49:46.315314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:49:46.315706 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:49:46.315929 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:46.316267 (+ 338us) server_negotiation.cc:299] Negotiation successful
0812 09:49:46.316535 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":176,"threads_started":1}
I20260812 09:49:46.319410 17480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7387a72c0b444d6ea88a911c8b52ae3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7387a72c0b444d6ea88a911c8b52ae3f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33271 } }
I20260812 09:49:46.319736 17480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7387a72c0b444d6ea88a911c8b52ae3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:49:46.319830 17480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7387a72c0b444d6ea88a911c8b52ae3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7387a72c0b444d6ea88a911c8b52ae3f, State: Initialized, Role: FOLLOWER
I20260812 09:49:46.320325 13708 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 09:49:46.320303 17480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7387a72c0b444d6ea88a911c8b52ae3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7387a72c0b444d6ea88a911c8b52ae3f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33271 } }
I20260812 09:49:46.320466 17480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7387a72c0b444d6ea88a911c8b52ae3f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 09:49:46.320551 17480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7387a72c0b444d6ea88a911c8b52ae3f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 09:49:46.320644 17480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7387a72c0b444d6ea88a911c8b52ae3f [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:49:46.321837 17480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7387a72c0b444d6ea88a911c8b52ae3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7387a72c0b444d6ea88a911c8b52ae3f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33271 } }
I20260812 09:49:46.322211 17480 leader_election.cc:302] T 00000000000000000000000000000000 P 7387a72c0b444d6ea88a911c8b52ae3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7387a72c0b444d6ea88a911c8b52ae3f; no voters:
I20260812 09:49:46.322657 17493 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7387a72c0b444d6ea88a911c8b52ae3f [term 1 FOLLOWER]: Leader election won for term 1
I20260812 09:49:46.322731 17480 leader_election.cc:288] T 00000000000000000000000000000000 P 7387a72c0b444d6ea88a911c8b52ae3f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 09:49:46.322893 17493 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7387a72c0b444d6ea88a911c8b52ae3f [term 1 LEADER]: Becoming Leader. State: Replica: 7387a72c0b444d6ea88a911c8b52ae3f, State: Running, Role: LEADER
I20260812 09:49:46.323203 17493 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7387a72c0b444d6ea88a911c8b52ae3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7387a72c0b444d6ea88a911c8b52ae3f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33271 } }
I20260812 09:49:46.323938 17480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7387a72c0b444d6ea88a911c8b52ae3f [sys.catalog]: configured and running, proceeding with master startup.
I20260812 09:49:46.324895 17498 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7387a72c0b444d6ea88a911c8b52ae3f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7387a72c0b444d6ea88a911c8b52ae3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7387a72c0b444d6ea88a911c8b52ae3f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33271 } } }
I20260812 09:49:46.325024 17498 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7387a72c0b444d6ea88a911c8b52ae3f [sys.catalog]: This master's current role is: LEADER
I20260812 09:49:46.324965 17497 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7387a72c0b444d6ea88a911c8b52ae3f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7387a72c0b444d6ea88a911c8b52ae3f. Latest consensus state: current_term: 1 leader_uuid: "7387a72c0b444d6ea88a911c8b52ae3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7387a72c0b444d6ea88a911c8b52ae3f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33271 } } }
I20260812 09:49:46.325088 17497 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7387a72c0b444d6ea88a911c8b52ae3f [sys.catalog]: This master's current role is: LEADER
W20260812 09:49:46.328912 17521 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7387a72c0b444d6ea88a911c8b52ae3f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 09:49:46.329010 17521 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 09:49:46.329090 17510 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 09:49:46.330116 17510 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 09:49:46.334889 17483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:46.322654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.48:42201 (local address 127.0.0.1:33271)
0812 09:49:46.322786 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:46.322790 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:46.323135 (+ 345us) server_negotiation.cc:407] Connection header received
0812 09:49:46.323301 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:46.323304 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:46.323346 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:46.323392 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:49:46.324280 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:46.325181 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:46.326538 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:46.326840 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:46.326899 (+ 59us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:46.328717 (+ 1818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:49:46.328721 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:49:46.328724 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:49:46.328749 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:49:46.332775 (+ 4026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:49:46.333350 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:49:46.333354 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:49:46.333356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:49:46.333448 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:49:46.333830 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:49:46.333833 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:49:46.333835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:49:46.334060 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:49:46.334210 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:46.334627 (+ 417us) server_negotiation.cc:299] Negotiation successful
0812 09:49:46.334731 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":51}
I20260812 09:49:46.336835 17510 catalog_manager.cc:1347] Generated new cluster ID: c081b329e4f1449ebca7584376d6c19e
I20260812 09:49:46.336908 17510 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 09:49:46.353739 17510 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 09:49:46.355001 17510 catalog_manager.cc:1504] Loading token signing keys...
I20260812 09:49:46.366094 17510 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7387a72c0b444d6ea88a911c8b52ae3f: Generated new TSK 0
I20260812 09:49:46.366781 17510 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 09:49:46.373826 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 17361
W20260812 09:49:46.387374 17339 connection.cc:535] client connection to 127.0.0.1:33271 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33271): Connection reset by peer (error 104)
2026-08-12T09:49:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1662 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786527905902985-13708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 09:49:46 dist-test-slave-jq4t krb5kdc[17531](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 09:49:46 dist-test-slave-jq4t krb5kdc[17531](info): set up 2 sockets
Aug 12 09:49:46 dist-test-slave-jq4t krb5kdc[17531](info): commencing operation
krb5kdc: starting...
W20260812 09:49:47.675272 13708 mini_kdc.cc:121] Time spent starting KDC: real 1.273s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786527905902985-13708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786527905902985-13708-0/krb5kdc/test-user.keytab.
Aug 12 09:49:47 dist-test-slave-jq4t krb5kdc[17531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528187, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T09:49:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T09:49:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786527905902985-13708-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786527905902985-13708-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 09:49:47.808298 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786527905902985-13708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786527905902985-13708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786527905902985-13708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41335
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786527905902985-13708-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41335
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:49:47.924746 17555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:49:47.925021 17555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:49:47.925093 17555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:49:47.929069 17555 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 09:49:47.929143 17555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:49:47.929178 17555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:49:47.929234 17555 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 09:49:47.929270 17555 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 09:49:47.934389 17555 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786527905902985-13708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786527905902985-13708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41335
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786527905902985-13708-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41335
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17555
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786527905902985-13708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:49:47.935395 17555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:49:47.936409 17555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:49:47.944835 17565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:49:47.944835 17566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:49:47.944972 17568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:49:47.945161 17555 server_base.cc:1034] running on GCE node
I20260812 09:49:47.945698 17555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:49:47.946740 17555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:49:47.947955 17555 hybrid_clock.cc:648] HybridClock initialized: now 1786528187947926 us; error 38 us; skew 500 ppm
Aug 12 09:49:47 dist-test-slave-jq4t krb5kdc[17531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528187, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:49:47.950786 17555 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 09:49:47.951681 17555 webserver.cc:460] Webserver started at http://127.0.0.1:46779/ using document root <none> and password file <none>
I20260812 09:49:47.952195 17555 fs_manager.cc:359] Metadata directory not provided
I20260812 09:49:47.952255 17555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:49:47.952419 17555 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:49:47.954528 17555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786527905902985-13708-0/minicluster-data/master-0/data/instance:
uuid: "bb8a64c7a9f44b0e81ab7300005f1e63"
format_stamp: "Formatted at 2026-08-12 09:49:47 on dist-test-slave-jq4t"
server_key: "8c1783dccd2e1c8b5cf1ea40472a9b13"
server_key_iv: "26fa689b05be0f60f1c2f6b1decfb0f3"
server_key_version: "encryptionkey@0"
I20260812 09:49:47.955092 17555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786527905902985-13708-0/minicluster-data/master-0/wal/instance:
uuid: "bb8a64c7a9f44b0e81ab7300005f1e63"
format_stamp: "Formatted at 2026-08-12 09:49:47 on dist-test-slave-jq4t"
server_key: "8c1783dccd2e1c8b5cf1ea40472a9b13"
server_key_iv: "26fa689b05be0f60f1c2f6b1decfb0f3"
server_key_version: "encryptionkey@0"
I20260812 09:49:47.959056 17555 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260812 09:49:47.961748 17579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:49:47.963186 17555 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 09:49:47.963356 17555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786527905902985-13708-0/minicluster-data/master-0/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786527905902985-13708-0/minicluster-data/master-0/wal
uuid: "bb8a64c7a9f44b0e81ab7300005f1e63"
format_stamp: "Formatted at 2026-08-12 09:49:47 on dist-test-slave-jq4t"
server_key: "8c1783dccd2e1c8b5cf1ea40472a9b13"
server_key_iv: "26fa689b05be0f60f1c2f6b1decfb0f3"
server_key_version: "encryptionkey@0"
I20260812 09:49:47.963477 17555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786527905902985-13708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786527905902985-13708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786527905902985-13708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:49:47.995503 17555 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:49:47.998693 17555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:49:47.998929 17555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:49:48.006752 17555 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41335
I20260812 09:49:48.006784 17659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41335 every 8 connection(s)
I20260812 09:49:48.007736 17555 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
I20260812 09:49:48.010761 17660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:49:48.015578 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 17555
I20260812 09:49:48.015715 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786527905902985-13708-0/minicluster-data/master-0/wal/instance
I20260812 09:49:48.016022 13708 external_mini_cluster.cc:1428] Setting key a63da9f6e70436a176dbc06a6d00b139
I20260812 09:49:48.017349 17660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb8a64c7a9f44b0e81ab7300005f1e63: Bootstrap starting.
I20260812 09:49:48.019764 17660 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb8a64c7a9f44b0e81ab7300005f1e63: Neither blocks nor log segments found. Creating new log.
I20260812 09:49:48.020589 17660 log.cc:824] T 00000000000000000000000000000000 P bb8a64c7a9f44b0e81ab7300005f1e63: Log is configured to *not* fsync() on all Append() calls
I20260812 09:49:48.022799 17660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb8a64c7a9f44b0e81ab7300005f1e63: No bootstrap required, opened a new log
I20260812 09:49:48.025162 17663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:48.017499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.48:46785 (local address 127.0.0.1:41335)
0812 09:49:48.017917 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:48.017933 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:48.017974 (+ 41us) server_negotiation.cc:407] Connection header received
0812 09:49:48.018597 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:48.018614 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:48.018942 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:48.019227 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:49:48.019598 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:48.020719 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:48.021826 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:48.022291 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:48.022409 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:48.023416 (+ 1007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:49:48.023422 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:49:48.023434 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 09:49:48.023818 (+ 384us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:49:48.023872 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 09:49:48.023893 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:49:48.024015 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:48.024249 (+ 234us) server_negotiation.cc:299] Negotiation successful
0812 09:49:48.024392 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":122,"threads_started":1}
I20260812 09:49:48.026650 17660 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb8a64c7a9f44b0e81ab7300005f1e63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb8a64c7a9f44b0e81ab7300005f1e63" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41335 } }
I20260812 09:49:48.026883 17660 raft_consensus.cc:374] T 00000000000000000000000000000000 P bb8a64c7a9f44b0e81ab7300005f1e63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:49:48.026991 17660 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb8a64c7a9f44b0e81ab7300005f1e63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb8a64c7a9f44b0e81ab7300005f1e63, State: Initialized, Role: FOLLOWER
I20260812 09:49:48.027420 17660 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb8a64c7a9f44b0e81ab7300005f1e63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb8a64c7a9f44b0e81ab7300005f1e63" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41335 } }
I20260812 09:49:48.027575 17660 raft_consensus.cc:388] T 00000000000000000000000000000000 P bb8a64c7a9f44b0e81ab7300005f1e63 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 09:49:48.027663 17660 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb8a64c7a9f44b0e81ab7300005f1e63 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 09:49:48.027778 17660 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb8a64c7a9f44b0e81ab7300005f1e63 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:49:48.027971 13708 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 09:49:48.028940 17660 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb8a64c7a9f44b0e81ab7300005f1e63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb8a64c7a9f44b0e81ab7300005f1e63" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41335 } }
I20260812 09:49:48.029309 17660 leader_election.cc:302] T 00000000000000000000000000000000 P bb8a64c7a9f44b0e81ab7300005f1e63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bb8a64c7a9f44b0e81ab7300005f1e63; no voters:
I20260812 09:49:48.029652 17660 leader_election.cc:288] T 00000000000000000000000000000000 P bb8a64c7a9f44b0e81ab7300005f1e63 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 09:49:48.029747 17674 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb8a64c7a9f44b0e81ab7300005f1e63 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 09:49:48.030498 17674 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb8a64c7a9f44b0e81ab7300005f1e63 [term 1 LEADER]: Becoming Leader. State: Replica: bb8a64c7a9f44b0e81ab7300005f1e63, State: Running, Role: LEADER
I20260812 09:49:48.030812 17674 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb8a64c7a9f44b0e81ab7300005f1e63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb8a64c7a9f44b0e81ab7300005f1e63" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41335 } }
I20260812 09:49:48.030961 17660 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb8a64c7a9f44b0e81ab7300005f1e63 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 09:49:48.032629 17675 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb8a64c7a9f44b0e81ab7300005f1e63 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bb8a64c7a9f44b0e81ab7300005f1e63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb8a64c7a9f44b0e81ab7300005f1e63" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41335 } } }
I20260812 09:49:48.032744 17675 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb8a64c7a9f44b0e81ab7300005f1e63 [sys.catalog]: This master's current role is: LEADER
I20260812 09:49:48.032394 17676 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb8a64c7a9f44b0e81ab7300005f1e63 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb8a64c7a9f44b0e81ab7300005f1e63. Latest consensus state: current_term: 1 leader_uuid: "bb8a64c7a9f44b0e81ab7300005f1e63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb8a64c7a9f44b0e81ab7300005f1e63" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41335 } } }
I20260812 09:49:48.033011 17676 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb8a64c7a9f44b0e81ab7300005f1e63 [sys.catalog]: This master's current role is: LEADER
W20260812 09:49:48.036059 17698 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bb8a64c7a9f44b0e81ab7300005f1e63: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 09:49:48.036192 17698 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 09:49:48.036548 17663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:48.030317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.48:48465 (local address 127.0.0.1:41335)
0812 09:49:48.030445 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:48.030449 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:48.032125 (+ 1676us) server_negotiation.cc:407] Connection header received
0812 09:49:48.032332 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:48.032337 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:48.032387 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:48.032437 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:49:48.032937 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:48.033680 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:48.034682 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:48.035015 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:48.035132 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:48.036012 (+ 880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:49:48.036016 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:49:48.036019 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 09:49:48.036032 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:49:48.036066 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 09:49:48.036076 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:49:48.036177 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:48.036355 (+ 178us) server_negotiation.cc:299] Negotiation successful
0812 09:49:48.036393 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260812 09:49:48.036676 17697 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 09:49:48.037554 17697 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 09:49:48.043092 17697 catalog_manager.cc:1347] Generated new cluster ID: f5a65e8d7a5c43a5963cc8460c2fd4be
I20260812 09:49:48.043162 17697 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 09:49:48.051842 17697 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 09:49:48.052919 17697 catalog_manager.cc:1504] Loading token signing keys...
I20260812 09:49:48.057058 17697 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bb8a64c7a9f44b0e81ab7300005f1e63: Generated new TSK 0
I20260812 09:49:48.057675 17697 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 09:49:48.062011 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 17555
W20260812 09:49:48.080777 17527 connection.cc:535] client connection to 127.0.0.1:41335 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41335): Connection reset by peer (error 104)
2026-08-12T09:49:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1693 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786527905902985-13708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 09:49:48 dist-test-slave-jq4t krb5kdc[17706](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 09:49:48 dist-test-slave-jq4t krb5kdc[17706](info): set up 2 sockets
Aug 12 09:49:48 dist-test-slave-jq4t krb5kdc[17706](info): commencing operation
krb5kdc: starting...
W20260812 09:49:49.349246 13708 mini_kdc.cc:121] Time spent starting KDC: real 1.254s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786527905902985-13708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786527905902985-13708-0/krb5kdc/test-user.keytab.
Aug 12 09:49:49 dist-test-slave-jq4t krb5kdc[17706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528189, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T09:49:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T09:49:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786527905902985-13708-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786527905902985-13708-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 09:49:49.489566 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786527905902985-13708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786527905902985-13708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786527905902985-13708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45891
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786527905902985-13708-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45891
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:49:49.605094 17726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:49:49.605443 17726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:49:49.605520 17726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:49:49.609690 17726 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 09:49:49.609783 17726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:49:49.609817 17726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:49:49.609854 17726 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 09:49:49.609889 17726 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 09:49:49.615430 17726 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786527905902985-13708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786527905902985-13708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45891
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786527905902985-13708-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17726
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786527905902985-13708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:49:49.616747 17726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:49:49.618069 17726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:49:49.626809 17737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:49:49.626825 17741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:49:49.626857 17735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:49:49.627070 17726 server_base.cc:1034] running on GCE node
I20260812 09:49:49.627754 17726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:49:49.629006 17726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:49:49.630223 17726 hybrid_clock.cc:648] HybridClock initialized: now 1786528189630208 us; error 28 us; skew 500 ppm
Aug 12 09:49:49 dist-test-slave-jq4t krb5kdc[17706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528189, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:49:49.633255 17726 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 09:49:49.634157 17726 webserver.cc:460] Webserver started at http://127.0.0.1:39039/ using document root <none> and password file <none>
I20260812 09:49:49.634701 17726 fs_manager.cc:359] Metadata directory not provided
I20260812 09:49:49.634763 17726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:49:49.634935 17726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:49:49.637131 17726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786527905902985-13708-0/minicluster-data/master-0/data/instance:
uuid: "b2f42d15d5114ff881b5ab2d6e15c6cd"
format_stamp: "Formatted at 2026-08-12 09:49:49 on dist-test-slave-jq4t"
server_key: "a17b338a5e3ebe067a26232b8d700746"
server_key_iv: "5c74dc1c3f0616aeea4bada779af5d48"
server_key_version: "encryptionkey@0"
I20260812 09:49:49.637670 17726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786527905902985-13708-0/minicluster-data/master-0/wal/instance:
uuid: "b2f42d15d5114ff881b5ab2d6e15c6cd"
format_stamp: "Formatted at 2026-08-12 09:49:49 on dist-test-slave-jq4t"
server_key: "a17b338a5e3ebe067a26232b8d700746"
server_key_iv: "5c74dc1c3f0616aeea4bada779af5d48"
server_key_version: "encryptionkey@0"
I20260812 09:49:49.641431 17726 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 09:49:49.643872 17757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:49:49.645182 17726 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 09:49:49.645292 17726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786527905902985-13708-0/minicluster-data/master-0/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786527905902985-13708-0/minicluster-data/master-0/wal
uuid: "b2f42d15d5114ff881b5ab2d6e15c6cd"
format_stamp: "Formatted at 2026-08-12 09:49:49 on dist-test-slave-jq4t"
server_key: "a17b338a5e3ebe067a26232b8d700746"
server_key_iv: "5c74dc1c3f0616aeea4bada779af5d48"
server_key_version: "encryptionkey@0"
I20260812 09:49:49.645377 17726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786527905902985-13708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786527905902985-13708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786527905902985-13708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:49:49.649957 17726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:49:49.657354 17726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:49:49.657511 17726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:49:49.665539 17726 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45891
I20260812 09:49:49.665573 17833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45891 every 8 connection(s)
I20260812 09:49:49.666575 17726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
I20260812 09:49:49.669435 17836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:49:49.675122 17836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2f42d15d5114ff881b5ab2d6e15c6cd: Bootstrap starting.
I20260812 09:49:49.676342 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 17726
I20260812 09:49:49.676527 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786527905902985-13708-0/minicluster-data/master-0/wal/instance
I20260812 09:49:49.676815 13708 external_mini_cluster.cc:1428] Setting key 8b5119a07414942c500c0901a75a2d6c
I20260812 09:49:49.677865 17836 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2f42d15d5114ff881b5ab2d6e15c6cd: Neither blocks nor log segments found. Creating new log.
I20260812 09:49:49.678712 17836 log.cc:824] T 00000000000000000000000000000000 P b2f42d15d5114ff881b5ab2d6e15c6cd: Log is configured to *not* fsync() on all Append() calls
I20260812 09:49:49.680859 17836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2f42d15d5114ff881b5ab2d6e15c6cd: No bootstrap required, opened a new log
I20260812 09:49:49.682184 17839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:49.677973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.48:54795 (local address 127.0.0.1:45891)
0812 09:49:49.678467 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:49.678482 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:49.678521 (+ 39us) server_negotiation.cc:407] Connection header received
0812 09:49:49.679418 (+ 897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:49.679446 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:49.679755 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:49.680109 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:49:49.680968 (+ 859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:49:49.680974 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:49:49.680991 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 09:49:49.681004 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:49:49.681070 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 09:49:49.681091 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:49:49.681213 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:49.681319 (+ 106us) server_negotiation.cc:299] Negotiation successful
0812 09:49:49.681500 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":146,"threads_started":1}
I20260812 09:49:49.685189 13708 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 09:49:49.685110 17836 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2f42d15d5114ff881b5ab2d6e15c6cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2f42d15d5114ff881b5ab2d6e15c6cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45891 } }
I20260812 09:49:49.685400 17836 raft_consensus.cc:374] T 00000000000000000000000000000000 P b2f42d15d5114ff881b5ab2d6e15c6cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:49:49.685477 17836 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2f42d15d5114ff881b5ab2d6e15c6cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2f42d15d5114ff881b5ab2d6e15c6cd, State: Initialized, Role: FOLLOWER
I20260812 09:49:49.685880 17836 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2f42d15d5114ff881b5ab2d6e15c6cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2f42d15d5114ff881b5ab2d6e15c6cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45891 } }
I20260812 09:49:49.686044 17836 raft_consensus.cc:388] T 00000000000000000000000000000000 P b2f42d15d5114ff881b5ab2d6e15c6cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 09:49:49.686134 17836 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2f42d15d5114ff881b5ab2d6e15c6cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 09:49:49.686237 17836 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2f42d15d5114ff881b5ab2d6e15c6cd [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:49:49.687222 17836 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2f42d15d5114ff881b5ab2d6e15c6cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2f42d15d5114ff881b5ab2d6e15c6cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45891 } }
I20260812 09:49:49.687598 17836 leader_election.cc:302] T 00000000000000000000000000000000 P b2f42d15d5114ff881b5ab2d6e15c6cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b2f42d15d5114ff881b5ab2d6e15c6cd; no voters:
I20260812 09:49:49.687975 17836 leader_election.cc:288] T 00000000000000000000000000000000 P b2f42d15d5114ff881b5ab2d6e15c6cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 09:49:49.688091 17851 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b2f42d15d5114ff881b5ab2d6e15c6cd [term 1 FOLLOWER]: Leader election won for term 1
I20260812 09:49:49.688325 17851 raft_consensus.cc:686] T 00000000000000000000000000000000 P b2f42d15d5114ff881b5ab2d6e15c6cd [term 1 LEADER]: Becoming Leader. State: Replica: b2f42d15d5114ff881b5ab2d6e15c6cd, State: Running, Role: LEADER
I20260812 09:49:49.688695 17851 consensus_queue.cc:237] T 00000000000000000000000000000000 P b2f42d15d5114ff881b5ab2d6e15c6cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2f42d15d5114ff881b5ab2d6e15c6cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45891 } }
I20260812 09:49:49.689236 17839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:49.687420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.48:33839 (local address 127.0.0.1:45891)
0812 09:49:49.687535 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:49.687539 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:49.687771 (+ 232us) server_negotiation.cc:407] Connection header received
0812 09:49:49.687892 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:49.687896 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:49.687927 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:49.688023 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:49:49.688814 (+ 791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:49:49.688818 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:49:49.688821 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 09:49:49.688833 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:49:49.688862 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 09:49:49.688874 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:49:49.688946 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:49.689070 (+ 124us) server_negotiation.cc:299] Negotiation successful
0812 09:49:49.689109 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260812 09:49:49.689304 17836 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2f42d15d5114ff881b5ab2d6e15c6cd [sys.catalog]: configured and running, proceeding with master startup.
I20260812 09:49:49.691077 17853 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2f42d15d5114ff881b5ab2d6e15c6cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2f42d15d5114ff881b5ab2d6e15c6cd. Latest consensus state: current_term: 1 leader_uuid: "b2f42d15d5114ff881b5ab2d6e15c6cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2f42d15d5114ff881b5ab2d6e15c6cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45891 } } }
I20260812 09:49:49.691205 17853 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2f42d15d5114ff881b5ab2d6e15c6cd [sys.catalog]: This master's current role is: LEADER
I20260812 09:49:49.691964 17852 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2f42d15d5114ff881b5ab2d6e15c6cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b2f42d15d5114ff881b5ab2d6e15c6cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2f42d15d5114ff881b5ab2d6e15c6cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45891 } } }
I20260812 09:49:49.692303 17852 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2f42d15d5114ff881b5ab2d6e15c6cd [sys.catalog]: This master's current role is: LEADER
W20260812 09:49:49.694521 17874 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b2f42d15d5114ff881b5ab2d6e15c6cd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 09:49:49.694633 17874 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 09:49:49.694742 17873 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 09:49:49.695658 17873 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 09:49:49.701020 17873 catalog_manager.cc:1347] Generated new cluster ID: 43e0627dad26437c823e2093041dceb3
I20260812 09:49:49.701083 17873 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 09:49:49.735750 17873 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 09:49:49.736135 17873 catalog_manager.cc:1504] Loading token signing keys...
I20260812 09:49:49.743430 17873 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b2f42d15d5114ff881b5ab2d6e15c6cd: Generated new TSK 0
I20260812 09:49:49.744150 17873 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 09:49:49.754475 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 17726
2026-08-12T09:49:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1687 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786527905902985-13708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 09:49:49 dist-test-slave-jq4t krb5kdc[17882](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 09:49:49 dist-test-slave-jq4t krb5kdc[17882](info): set up 2 sockets
Aug 12 09:49:49 dist-test-slave-jq4t krb5kdc[17882](info): commencing operation
krb5kdc: starting...
W20260812 09:49:51.034776 13708 mini_kdc.cc:121] Time spent starting KDC: real 1.252s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786527905902985-13708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786527905902985-13708-0/krb5kdc/test-user.keytab.
Aug 12 09:49:51 dist-test-slave-jq4t krb5kdc[17882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T09:49:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T09:49:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.99.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.99.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.99.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab.
Entry for principal kudu/127.13.99.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab.
I20260812 09:49:51.163635 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786527905902985-13708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786527905902985-13708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786527905902985-13708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.62:39071
--webserver_interface=127.13.99.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab
--principal=kudu/127.13.99.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:38493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.99.62:39071
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:49:51.281380 17909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:49:51.281677 17909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:49:51.281720 17909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:49:51.285852 17909 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 09:49:51.285912 17909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:49:51.285933 17909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:49:51.285949 17909 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 09:49:51.285964 17909 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 09:49:51.291361 17909 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:38493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786527905902985-13708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786527905902985-13708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.99.62:39071
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab
--principal=kudu/127.13.99.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.62:39071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.99.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17909
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786527905902985-13708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:49:51.292557 17909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:49:51.293642 17909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:49:51.302393 17924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:49:51.302386 17920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:49:51.302386 17921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:49:51.302778 17909 server_base.cc:1034] running on GCE node
I20260812 09:49:51.303489 17909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:49:51.304646 17909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:49:51.305825 17909 hybrid_clock.cc:648] HybridClock initialized: now 1786528191305796 us; error 40 us; skew 500 ppm
Aug 12 09:49:51 dist-test-slave-jq4t krb5kdc[17882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528191, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:49:51.308733 17909 init.cc:377] Logged in from keytab as kudu/127.13.99.62@KRBTEST.COM (short username kudu)
I20260812 09:49:51.309659 17909 webserver.cc:460] Webserver started at http://127.13.99.62:39603/ using document root <none> and password file <none>
I20260812 09:49:51.310240 17909 fs_manager.cc:359] Metadata directory not provided
I20260812 09:49:51.310333 17909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:49:51.310576 17909 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:49:51.312716 17909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786527905902985-13708-0/minicluster-data/master-0/data/instance:
uuid: "1b13a713e3b046b7a5eb2d32aa82fdb9"
format_stamp: "Formatted at 2026-08-12 09:49:51 on dist-test-slave-jq4t"
server_key: "b22747c90882cc2ffd498a31efc60385"
server_key_iv: "a8a4f7122903acc21a2b96f0e18ce9e6"
server_key_version: "encryptionkey@0"
I20260812 09:49:51.313308 17909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786527905902985-13708-0/minicluster-data/master-0/wal/instance:
uuid: "1b13a713e3b046b7a5eb2d32aa82fdb9"
format_stamp: "Formatted at 2026-08-12 09:49:51 on dist-test-slave-jq4t"
server_key: "b22747c90882cc2ffd498a31efc60385"
server_key_iv: "a8a4f7122903acc21a2b96f0e18ce9e6"
server_key_version: "encryptionkey@0"
I20260812 09:49:51.317096 17909 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 09:49:51.319630 17935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:49:51.320959 17909 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 09:49:51.321123 17909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786527905902985-13708-0/minicluster-data/master-0/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786527905902985-13708-0/minicluster-data/master-0/wal
uuid: "1b13a713e3b046b7a5eb2d32aa82fdb9"
format_stamp: "Formatted at 2026-08-12 09:49:51 on dist-test-slave-jq4t"
server_key: "b22747c90882cc2ffd498a31efc60385"
server_key_iv: "a8a4f7122903acc21a2b96f0e18ce9e6"
server_key_version: "encryptionkey@0"
I20260812 09:49:51.321240 17909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786527905902985-13708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786527905902985-13708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786527905902985-13708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:49:51.337437 17909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:49:51.340798 17909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:49:51.341018 17909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:49:51.348450 17909 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.62:39071
I20260812 09:49:51.348510 18032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.62:39071 every 8 connection(s)
I20260812 09:49:51.349468 17909 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
I20260812 09:49:51.351073 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 17909
I20260812 09:49:51.351228 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786527905902985-13708-0/minicluster-data/master-0/wal/instance
I20260812 09:49:51.351544 13708 external_mini_cluster.cc:1428] Setting key 980d6de322a8e605d763a01bc5ec29af
I20260812 09:49:51.352722 18033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:49:51.358438 18033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b13a713e3b046b7a5eb2d32aa82fdb9: Bootstrap starting.
Aug 12 09:49:51 dist-test-slave-jq4t krb5kdc[17882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
I20260812 09:49:51.360960 18033 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b13a713e3b046b7a5eb2d32aa82fdb9: Neither blocks nor log segments found. Creating new log.
I20260812 09:49:51.361915 18033 log.cc:824] T 00000000000000000000000000000000 P 1b13a713e3b046b7a5eb2d32aa82fdb9: Log is configured to *not* fsync() on all Append() calls
I20260812 09:49:51.364354 18033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b13a713e3b046b7a5eb2d32aa82fdb9: No bootstrap required, opened a new log
I20260812 09:49:51.366852 18039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:51.353063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52595 (local address 127.13.99.62:39071)
0812 09:49:51.353525 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:51.353541 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:51.353578 (+ 37us) server_negotiation.cc:407] Connection header received
0812 09:49:51.354243 (+ 665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:51.354260 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:51.354591 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:51.354942 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:49:51.355766 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:51.356855 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:51.357740 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:51.358256 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:51.358429 (+ 173us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:51.361326 (+ 2897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:49:51.361333 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:49:51.361350 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:49:51.361383 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:49:51.363503 (+ 2120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:49:51.364046 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:49:51.364053 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:49:51.364058 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:49:51.364198 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:49:51.364609 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:49:51.364612 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:49:51.364614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:49:51.365141 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:49:51.365358 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:51.365613 (+ 255us) server_negotiation.cc:299] Negotiation successful
0812 09:49:51.365835 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":156,"threads_started":1}
I20260812 09:49:51.368913 18033 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b13a713e3b046b7a5eb2d32aa82fdb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b13a713e3b046b7a5eb2d32aa82fdb9" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 39071 } }
I20260812 09:49:51.369179 18033 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1b13a713e3b046b7a5eb2d32aa82fdb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:49:51.369243 18033 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b13a713e3b046b7a5eb2d32aa82fdb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b13a713e3b046b7a5eb2d32aa82fdb9, State: Initialized, Role: FOLLOWER
I20260812 09:49:51.369732 18033 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b13a713e3b046b7a5eb2d32aa82fdb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b13a713e3b046b7a5eb2d32aa82fdb9" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 39071 } }
I20260812 09:49:51.369896 18033 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1b13a713e3b046b7a5eb2d32aa82fdb9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 09:49:51.369961 18033 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b13a713e3b046b7a5eb2d32aa82fdb9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 09:49:51.370110 18033 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b13a713e3b046b7a5eb2d32aa82fdb9 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:49:51.370239 13708 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 09:49:51.371116 18033 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b13a713e3b046b7a5eb2d32aa82fdb9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b13a713e3b046b7a5eb2d32aa82fdb9" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 39071 } }
I20260812 09:49:51.371469 18033 leader_election.cc:302] T 00000000000000000000000000000000 P 1b13a713e3b046b7a5eb2d32aa82fdb9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b13a713e3b046b7a5eb2d32aa82fdb9; no voters:
I20260812 09:49:51.371789 18033 leader_election.cc:288] T 00000000000000000000000000000000 P 1b13a713e3b046b7a5eb2d32aa82fdb9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 09:49:51.371886 18050 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b13a713e3b046b7a5eb2d32aa82fdb9 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 09:49:51.372133 18050 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b13a713e3b046b7a5eb2d32aa82fdb9 [term 1 LEADER]: Becoming Leader. State: Replica: 1b13a713e3b046b7a5eb2d32aa82fdb9, State: Running, Role: LEADER
I20260812 09:49:51.372650 18050 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b13a713e3b046b7a5eb2d32aa82fdb9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b13a713e3b046b7a5eb2d32aa82fdb9" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 39071 } }
I20260812 09:49:51.373159 18033 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b13a713e3b046b7a5eb2d32aa82fdb9 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 09:49:51.374507 18051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b13a713e3b046b7a5eb2d32aa82fdb9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1b13a713e3b046b7a5eb2d32aa82fdb9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b13a713e3b046b7a5eb2d32aa82fdb9" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 39071 } } }
I20260812 09:49:51.374636 18051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b13a713e3b046b7a5eb2d32aa82fdb9 [sys.catalog]: This master's current role is: LEADER
I20260812 09:49:51.374974 18053 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b13a713e3b046b7a5eb2d32aa82fdb9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b13a713e3b046b7a5eb2d32aa82fdb9. Latest consensus state: current_term: 1 leader_uuid: "1b13a713e3b046b7a5eb2d32aa82fdb9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b13a713e3b046b7a5eb2d32aa82fdb9" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 39071 } } }
I20260812 09:49:51.375038 18065 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 09:49:51.375073 18053 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b13a713e3b046b7a5eb2d32aa82fdb9 [sys.catalog]: This master's current role is: LEADER
I20260812 09:49:51.378307 18065 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 09:49:51.384047 18039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:51.372421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36555 (local address 127.13.99.62:39071)
0812 09:49:51.372549 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:51.372553 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:51.373023 (+ 470us) server_negotiation.cc:407] Connection header received
0812 09:49:51.373125 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:51.373128 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:51.373176 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:51.373259 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:49:51.375787 (+ 2528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:51.376383 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:51.377586 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:51.377936 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:51.378049 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:51.379779 (+ 1730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:49:51.379783 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:49:51.379786 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:49:51.379812 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:49:51.381597 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:49:51.382186 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:49:51.382190 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:49:51.382192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:49:51.382281 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:49:51.382681 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:49:51.382685 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:49:51.382686 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:49:51.382901 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:49:51.383043 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:51.383851 (+ 808us) server_negotiation.cc:299] Negotiation successful
0812 09:49:51.383938 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260812 09:49:51.386016 18065 catalog_manager.cc:1347] Generated new cluster ID: 6298ec7aa835453b8fb11783f53039f3
I20260812 09:49:51.386080 18065 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 09:49:51.420763 18065 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 09:49:51.421769 18065 catalog_manager.cc:1504] Loading token signing keys...
I20260812 09:49:51.429737 18065 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1b13a713e3b046b7a5eb2d32aa82fdb9: Generated new TSK 0
I20260812 09:49:51.430709 18065 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 09:49:51.437707 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 17909
W20260812 09:49:51.450538 17879 connection.cc:535] client connection to 127.13.99.62:39071 recv error: Network error: failed to read from TLS socket (remote: 127.13.99.62:39071): Connection reset by peer (error 104)
2026-08-12T09:49:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1682 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786527905902985-13708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 09:49:51 dist-test-slave-jq4t krb5kdc[18088](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 09:49:51 dist-test-slave-jq4t krb5kdc[18088](info): set up 2 sockets
Aug 12 09:49:51 dist-test-slave-jq4t krb5kdc[18088](info): commencing operation
krb5kdc: starting...
W20260812 09:49:52.726104 13708 mini_kdc.cc:121] Time spent starting KDC: real 1.261s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786527905902985-13708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786527905902985-13708-0/krb5kdc/test-user.keytab.
Aug 12 09:49:52 dist-test-slave-jq4t krb5kdc[18088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528192, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T09:49:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T09:49:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.99.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.99.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.99.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab.
Entry for principal kudu/127.13.99.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab.
I20260812 09:49:52.857609 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786527905902985-13708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786527905902985-13708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786527905902985-13708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.62:42423
--webserver_interface=127.13.99.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab
--principal=kudu/127.13.99.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:36463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.99.62:42423
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:49:52.969029 18104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:49:52.969318 18104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:49:52.969400 18104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:49:52.973532 18104 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 09:49:52.973614 18104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:49:52.973654 18104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:49:52.973711 18104 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 09:49:52.973750 18104 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 09:49:52.978781 18104 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:36463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786527905902985-13708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786527905902985-13708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.99.62:42423
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab
--principal=kudu/127.13.99.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.62:42423
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.99.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786527905902985-13708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:49:52.979759 18104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:49:52.980782 18104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:49:52.989378 18117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:49:52.989378 18116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:49:52.989506 18121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:49:52.989642 18104 server_base.cc:1034] running on GCE node
I20260812 09:49:52.990265 18104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:49:52.991348 18104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:49:52.992580 18104 hybrid_clock.cc:648] HybridClock initialized: now 1786528192992561 us; error 31 us; skew 500 ppm
Aug 12 09:49:52 dist-test-slave-jq4t krb5kdc[18088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528192, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:49:52.995499 18104 init.cc:377] Logged in from keytab as kudu/127.13.99.62@KRBTEST.COM (short username kudu)
I20260812 09:49:52.996443 18104 webserver.cc:460] Webserver started at http://127.13.99.62:39439/ using document root <none> and password file <none>
I20260812 09:49:52.997046 18104 fs_manager.cc:359] Metadata directory not provided
I20260812 09:49:52.997144 18104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:49:52.997390 18104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:49:52.999439 18104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786527905902985-13708-0/minicluster-data/master-0/data/instance:
uuid: "1222daf101d0456b965ae1694f5f4b4d"
format_stamp: "Formatted at 2026-08-12 09:49:52 on dist-test-slave-jq4t"
server_key: "c46b8ba3062ab40f7dae27fd0050bc5f"
server_key_iv: "fd10c82567373d58a355fd1ac8b10f5d"
server_key_version: "encryptionkey@0"
I20260812 09:49:53.000036 18104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786527905902985-13708-0/minicluster-data/master-0/wal/instance:
uuid: "1222daf101d0456b965ae1694f5f4b4d"
format_stamp: "Formatted at 2026-08-12 09:49:52 on dist-test-slave-jq4t"
server_key: "c46b8ba3062ab40f7dae27fd0050bc5f"
server_key_iv: "fd10c82567373d58a355fd1ac8b10f5d"
server_key_version: "encryptionkey@0"
I20260812 09:49:53.003935 18104 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260812 09:49:53.006691 18132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:49:53.007968 18104 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 09:49:53.008149 18104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786527905902985-13708-0/minicluster-data/master-0/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786527905902985-13708-0/minicluster-data/master-0/wal
uuid: "1222daf101d0456b965ae1694f5f4b4d"
format_stamp: "Formatted at 2026-08-12 09:49:52 on dist-test-slave-jq4t"
server_key: "c46b8ba3062ab40f7dae27fd0050bc5f"
server_key_iv: "fd10c82567373d58a355fd1ac8b10f5d"
server_key_version: "encryptionkey@0"
I20260812 09:49:53.008271 18104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786527905902985-13708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786527905902985-13708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786527905902985-13708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:49:53.043818 18104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:49:53.047663 18104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:49:53.047910 18104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:49:53.056356 18104 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.62:42423
I20260812 09:49:53.056367 18224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.62:42423 every 8 connection(s)
I20260812 09:49:53.057439 18104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
I20260812 09:49:53.060271 18225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:49:53.064095 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 18104
I20260812 09:49:53.064256 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786527905902985-13708-0/minicluster-data/master-0/wal/instance
I20260812 09:49:53.064580 13708 external_mini_cluster.cc:1428] Setting key ee41a1892c009e2557840dd72a7a9675
I20260812 09:49:53.066329 18225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1222daf101d0456b965ae1694f5f4b4d: Bootstrap starting.
I20260812 09:49:53.068742 18225 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1222daf101d0456b965ae1694f5f4b4d: Neither blocks nor log segments found. Creating new log.
I20260812 09:49:53.069561 18225 log.cc:824] T 00000000000000000000000000000000 P 1222daf101d0456b965ae1694f5f4b4d: Log is configured to *not* fsync() on all Append() calls
I20260812 09:49:53.071745 18225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1222daf101d0456b965ae1694f5f4b4d: No bootstrap required, opened a new log
I20260812 09:49:53.073912 18228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:53.065990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36433 (local address 127.13.99.62:42423)
0812 09:49:53.066405 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:53.066427 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:53.066475 (+ 48us) server_negotiation.cc:407] Connection header received
0812 09:49:53.067113 (+ 638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:53.067131 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:53.067451 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:53.067719 (+ 268us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:49:53.068110 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:53.069242 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:53.070334 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:53.070924 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:53.071053 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:53.071959 (+ 906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:49:53.071964 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:49:53.071993 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 09:49:53.072366 (+ 373us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:49:53.072421 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 09:49:53.072444 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:49:53.072622 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:53.072813 (+ 191us) server_negotiation.cc:299] Negotiation successful
0812 09:49:53.072969 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":135,"threads_started":1}
I20260812 09:49:53.074978 18225 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1222daf101d0456b965ae1694f5f4b4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1222daf101d0456b965ae1694f5f4b4d" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 42423 } }
I20260812 09:49:53.075244 18225 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1222daf101d0456b965ae1694f5f4b4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:49:53.075307 18225 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1222daf101d0456b965ae1694f5f4b4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1222daf101d0456b965ae1694f5f4b4d, State: Initialized, Role: FOLLOWER
I20260812 09:49:53.075762 18225 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1222daf101d0456b965ae1694f5f4b4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1222daf101d0456b965ae1694f5f4b4d" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 42423 } }
I20260812 09:49:53.075945 18225 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1222daf101d0456b965ae1694f5f4b4d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 09:49:53.076069 18225 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1222daf101d0456b965ae1694f5f4b4d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 09:49:53.076186 18225 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1222daf101d0456b965ae1694f5f4b4d [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:49:53.077214 13708 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 09:49:53.077349 18225 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1222daf101d0456b965ae1694f5f4b4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1222daf101d0456b965ae1694f5f4b4d" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 42423 } }
I20260812 09:49:53.077709 18225 leader_election.cc:302] T 00000000000000000000000000000000 P 1222daf101d0456b965ae1694f5f4b4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1222daf101d0456b965ae1694f5f4b4d; no voters:
I20260812 09:49:53.078058 18225 leader_election.cc:288] T 00000000000000000000000000000000 P 1222daf101d0456b965ae1694f5f4b4d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 09:49:53.078167 18231 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1222daf101d0456b965ae1694f5f4b4d [term 1 FOLLOWER]: Leader election won for term 1
I20260812 09:49:53.078393 18231 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1222daf101d0456b965ae1694f5f4b4d [term 1 LEADER]: Becoming Leader. State: Replica: 1222daf101d0456b965ae1694f5f4b4d, State: Running, Role: LEADER
I20260812 09:49:53.078749 18231 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1222daf101d0456b965ae1694f5f4b4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1222daf101d0456b965ae1694f5f4b4d" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 42423 } }
I20260812 09:49:53.079420 18225 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1222daf101d0456b965ae1694f5f4b4d [sys.catalog]: configured and running, proceeding with master startup.
I20260812 09:49:53.080257 18234 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1222daf101d0456b965ae1694f5f4b4d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1222daf101d0456b965ae1694f5f4b4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1222daf101d0456b965ae1694f5f4b4d" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 42423 } } }
I20260812 09:49:53.080391 18234 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1222daf101d0456b965ae1694f5f4b4d [sys.catalog]: This master's current role is: LEADER
I20260812 09:49:53.080626 18235 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1222daf101d0456b965ae1694f5f4b4d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1222daf101d0456b965ae1694f5f4b4d. Latest consensus state: current_term: 1 leader_uuid: "1222daf101d0456b965ae1694f5f4b4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1222daf101d0456b965ae1694f5f4b4d" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 42423 } } }
I20260812 09:49:53.080785 18235 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1222daf101d0456b965ae1694f5f4b4d [sys.catalog]: This master's current role is: LEADER
I20260812 09:49:53.080876 18255 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 09:49:53.084125 18255 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 09:49:53.087271 18228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:53.079538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47089 (local address 127.13.99.62:42423)
0812 09:49:53.079650 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:53.079656 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:53.082314 (+ 2658us) server_negotiation.cc:407] Connection header received
0812 09:49:53.082475 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:53.082478 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:53.082525 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:53.082599 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:49:53.082919 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:53.083673 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:53.085378 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:53.085692 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:53.085807 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:53.086765 (+ 958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:49:53.086769 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:49:53.086772 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 09:49:53.086785 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:49:53.086814 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 09:49:53.086827 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:49:53.086919 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:53.087110 (+ 191us) server_negotiation.cc:299] Negotiation successful
0812 09:49:53.087146 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260812 09:49:53.090287 18255 catalog_manager.cc:1347] Generated new cluster ID: 3c2affdbb1094c09b18216654dd90891
I20260812 09:49:53.090349 18255 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 09:49:53.106638 18255 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 09:49:53.107847 18255 catalog_manager.cc:1504] Loading token signing keys...
I20260812 09:49:53.126152 18255 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1222daf101d0456b965ae1694f5f4b4d: Generated new TSK 0
I20260812 09:49:53.126856 18255 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 09:49:53.135217 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 18104
W20260812 09:49:53.148918 18083 connection.cc:535] client connection to 127.13.99.62:42423 recv error: Network error: failed to read from TLS socket (remote: 127.13.99.62:42423): Connection reset by peer (error 104)
2026-08-12T09:49:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1698 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786527905902985-13708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 09:49:53 dist-test-slave-jq4t krb5kdc[18280](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 09:49:53 dist-test-slave-jq4t krb5kdc[18280](info): set up 2 sockets
Aug 12 09:49:53 dist-test-slave-jq4t krb5kdc[18280](info): commencing operation
krb5kdc: starting...
W20260812 09:49:54.426029 13708 mini_kdc.cc:121] Time spent starting KDC: real 1.263s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786527905902985-13708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786527905902985-13708-0/krb5kdc/test-user.keytab.
Aug 12 09:49:54 dist-test-slave-jq4t krb5kdc[18280](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528194, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T09:49:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T09:49:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.99.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.99.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.99.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab.
Entry for principal kudu/127.13.99.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab.
I20260812 09:49:54.561966 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786527905902985-13708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786527905902985-13708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786527905902985-13708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.62:41769
--webserver_interface=127.13.99.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab
--principal=kudu/127.13.99.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:36587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.99.62:41769
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:49:54.677388 18298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:49:54.677660 18298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:49:54.677736 18298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:49:54.682463 18298 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 09:49:54.682539 18298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:49:54.682572 18298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:49:54.682596 18298 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 09:49:54.682629 18298 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 09:49:54.688167 18298 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:36587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786527905902985-13708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786527905902985-13708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.99.62:41769
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab
--principal=kudu/127.13.99.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.62:41769
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.99.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18298
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786527905902985-13708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:49:54.689261 18298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:49:54.690272 18298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:49:54.698453 18307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:49:54.698505 18312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:49:54.698485 18309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:49:54.698689 18298 server_base.cc:1034] running on GCE node
I20260812 09:49:54.699280 18298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:49:54.700373 18298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:49:54.701603 18298 hybrid_clock.cc:648] HybridClock initialized: now 1786528194701579 us; error 35 us; skew 500 ppm
Aug 12 09:49:54 dist-test-slave-jq4t krb5kdc[18280](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528194, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:49:54.708763 18298 init.cc:377] Logged in from keytab as kudu/127.13.99.62@KRBTEST.COM (short username kudu)
I20260812 09:49:54.709659 18298 webserver.cc:460] Webserver started at http://127.13.99.62:42383/ using document root <none> and password file <none>
I20260812 09:49:54.710188 18298 fs_manager.cc:359] Metadata directory not provided
I20260812 09:49:54.710249 18298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:49:54.710422 18298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:49:54.712447 18298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786527905902985-13708-0/minicluster-data/master-0/data/instance:
uuid: "165782bf29cf46a5ba9e140aa016772f"
format_stamp: "Formatted at 2026-08-12 09:49:54 on dist-test-slave-jq4t"
server_key: "542f9c872947a963472596179629bc35"
server_key_iv: "00550923c9f73aae7818d6d51fd775fb"
server_key_version: "encryptionkey@0"
I20260812 09:49:54.713085 18298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786527905902985-13708-0/minicluster-data/master-0/wal/instance:
uuid: "165782bf29cf46a5ba9e140aa016772f"
format_stamp: "Formatted at 2026-08-12 09:49:54 on dist-test-slave-jq4t"
server_key: "542f9c872947a963472596179629bc35"
server_key_iv: "00550923c9f73aae7818d6d51fd775fb"
server_key_version: "encryptionkey@0"
I20260812 09:49:54.716919 18298 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 09:49:54.719471 18323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:49:54.720853 18298 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 09:49:54.720991 18298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786527905902985-13708-0/minicluster-data/master-0/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786527905902985-13708-0/minicluster-data/master-0/wal
uuid: "165782bf29cf46a5ba9e140aa016772f"
format_stamp: "Formatted at 2026-08-12 09:49:54 on dist-test-slave-jq4t"
server_key: "542f9c872947a963472596179629bc35"
server_key_iv: "00550923c9f73aae7818d6d51fd775fb"
server_key_version: "encryptionkey@0"
I20260812 09:49:54.721097 18298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786527905902985-13708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786527905902985-13708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786527905902985-13708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:49:54.725760 18298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:49:54.728657 18298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:49:54.728842 18298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:49:54.736061 18298 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.62:41769
I20260812 09:49:54.736083 18408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.62:41769 every 8 connection(s)
I20260812 09:49:54.737072 18298 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
I20260812 09:49:54.739681 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 18298
I20260812 09:49:54.739825 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786527905902985-13708-0/minicluster-data/master-0/wal/instance
I20260812 09:49:54.739921 18410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:49:54.740159 13708 external_mini_cluster.cc:1428] Setting key 7e05b6ad036d83496d0fbc3dbc03961f
I20260812 09:49:54.745579 18415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:54.741573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60585 (local address 127.13.99.62:41769)
0812 09:49:54.742047 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:54.742062 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:54.742096 (+ 34us) server_negotiation.cc:407] Connection header received
0812 09:49:54.743011 (+ 915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:54.743031 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:54.743302 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:54.743636 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:49:54.744413 (+ 777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:49:54.744418 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:49:54.744429 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 09:49:54.744440 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:49:54.744536 (+ 96us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 09:49:54.744572 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:49:54.744724 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:54.744795 (+ 71us) server_negotiation.cc:299] Negotiation successful
0812 09:49:54.744958 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":149,"threads_started":1}
I20260812 09:49:54.746383 18410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 165782bf29cf46a5ba9e140aa016772f: Bootstrap starting.
I20260812 09:49:54.748214 13708 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 09:49:54.748754 18410 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 165782bf29cf46a5ba9e140aa016772f: Neither blocks nor log segments found. Creating new log.
I20260812 09:49:54.749507 18410 log.cc:824] T 00000000000000000000000000000000 P 165782bf29cf46a5ba9e140aa016772f: Log is configured to *not* fsync() on all Append() calls
I20260812 09:49:54.751466 18410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 165782bf29cf46a5ba9e140aa016772f: No bootstrap required, opened a new log
I20260812 09:49:54.751947 18415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:54.750277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51683 (local address 127.13.99.62:41769)
0812 09:49:54.750369 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:54.750373 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:54.750574 (+ 201us) server_negotiation.cc:407] Connection header received
0812 09:49:54.750722 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:54.750725 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:54.750751 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:54.750847 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:49:54.751570 (+ 723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:49:54.751573 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:49:54.751575 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 09:49:54.751587 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:49:54.751623 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 09:49:54.751633 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:49:54.751720 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:54.751797 (+ 77us) server_negotiation.cc:299] Negotiation successful
0812 09:49:54.751834 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260812 09:49:54.755277 18410 raft_consensus.cc:348] T 00000000000000000000000000000000 P 165782bf29cf46a5ba9e140aa016772f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "165782bf29cf46a5ba9e140aa016772f" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 41769 } }
I20260812 09:49:54.755522 18410 raft_consensus.cc:374] T 00000000000000000000000000000000 P 165782bf29cf46a5ba9e140aa016772f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:49:54.755607 18410 raft_consensus.cc:729] T 00000000000000000000000000000000 P 165782bf29cf46a5ba9e140aa016772f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 165782bf29cf46a5ba9e140aa016772f, State: Initialized, Role: FOLLOWER
I20260812 09:49:54.756029 18410 consensus_queue.cc:260] T 00000000000000000000000000000000 P 165782bf29cf46a5ba9e140aa016772f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "165782bf29cf46a5ba9e140aa016772f" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 41769 } }
I20260812 09:49:54.756179 18410 raft_consensus.cc:388] T 00000000000000000000000000000000 P 165782bf29cf46a5ba9e140aa016772f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 09:49:54.756269 18410 raft_consensus.cc:482] T 00000000000000000000000000000000 P 165782bf29cf46a5ba9e140aa016772f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 09:49:54.756369 18410 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 165782bf29cf46a5ba9e140aa016772f [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:49:54.757346 18410 raft_consensus.cc:504] T 00000000000000000000000000000000 P 165782bf29cf46a5ba9e140aa016772f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "165782bf29cf46a5ba9e140aa016772f" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 41769 } }
I20260812 09:49:54.757676 18410 leader_election.cc:302] T 00000000000000000000000000000000 P 165782bf29cf46a5ba9e140aa016772f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 165782bf29cf46a5ba9e140aa016772f; no voters:
I20260812 09:49:54.757977 18410 leader_election.cc:288] T 00000000000000000000000000000000 P 165782bf29cf46a5ba9e140aa016772f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 09:49:54.758100 18423 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 165782bf29cf46a5ba9e140aa016772f [term 1 FOLLOWER]: Leader election won for term 1
I20260812 09:49:54.758340 18423 raft_consensus.cc:686] T 00000000000000000000000000000000 P 165782bf29cf46a5ba9e140aa016772f [term 1 LEADER]: Becoming Leader. State: Replica: 165782bf29cf46a5ba9e140aa016772f, State: Running, Role: LEADER
I20260812 09:49:54.758628 18423 consensus_queue.cc:237] T 00000000000000000000000000000000 P 165782bf29cf46a5ba9e140aa016772f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "165782bf29cf46a5ba9e140aa016772f" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 41769 } }
I20260812 09:49:54.759209 18410 sys_catalog.cc:564] T 00000000000000000000000000000000 P 165782bf29cf46a5ba9e140aa016772f [sys.catalog]: configured and running, proceeding with master startup.
I20260812 09:49:54.759810 18428 sys_catalog.cc:455] T 00000000000000000000000000000000 P 165782bf29cf46a5ba9e140aa016772f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 165782bf29cf46a5ba9e140aa016772f. Latest consensus state: current_term: 1 leader_uuid: "165782bf29cf46a5ba9e140aa016772f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "165782bf29cf46a5ba9e140aa016772f" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 41769 } } }
I20260812 09:49:54.759944 18428 sys_catalog.cc:458] T 00000000000000000000000000000000 P 165782bf29cf46a5ba9e140aa016772f [sys.catalog]: This master's current role is: LEADER
I20260812 09:49:54.760219 18426 sys_catalog.cc:455] T 00000000000000000000000000000000 P 165782bf29cf46a5ba9e140aa016772f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "165782bf29cf46a5ba9e140aa016772f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "165782bf29cf46a5ba9e140aa016772f" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 41769 } } }
I20260812 09:49:54.760326 18426 sys_catalog.cc:458] T 00000000000000000000000000000000 P 165782bf29cf46a5ba9e140aa016772f [sys.catalog]: This master's current role is: LEADER
I20260812 09:49:54.760350 18438 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 09:49:54.763612 18438 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 09:49:54.769127 18438 catalog_manager.cc:1347] Generated new cluster ID: 9f564343a54b4594858e5170b888213a
I20260812 09:49:54.769201 18438 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 09:49:54.784886 18438 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 09:49:54.785216 18438 catalog_manager.cc:1504] Loading token signing keys...
I20260812 09:49:54.791671 18438 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 165782bf29cf46a5ba9e140aa016772f: Generated new TSK 0
I20260812 09:49:54.792325 18438 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 09:49:54.803826 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 18298
2026-08-12T09:49:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1675 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786527905902985-13708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 09:49:54 dist-test-slave-jq4t krb5kdc[18460](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 09:49:54 dist-test-slave-jq4t krb5kdc[18460](info): set up 2 sockets
Aug 12 09:49:54 dist-test-slave-jq4t krb5kdc[18460](info): commencing operation
krb5kdc: starting...
W20260812 09:49:56.087738 13708 mini_kdc.cc:121] Time spent starting KDC: real 1.249s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786527905902985-13708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786527905902985-13708-0/krb5kdc/test-user.keytab.
Aug 12 09:49:56 dist-test-slave-jq4t krb5kdc[18460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528196, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T09:49:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T09:49:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.99.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.99.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.99.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab.
Entry for principal kudu/127.13.99.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab.
I20260812 09:49:56.216662 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786527905902985-13708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786527905902985-13708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786527905902985-13708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.62:35949
--webserver_interface=127.13.99.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab
--principal=kudu/127.13.99.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:34529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.99.62:35949
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:49:56.331391 18482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:49:56.331720 18482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:49:56.331804 18482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:49:56.336009 18482 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 09:49:56.336109 18482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:49:56.336149 18482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:49:56.336187 18482 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 09:49:56.336226 18482 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 09:49:56.341825 18482 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:34529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786527905902985-13708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786527905902985-13708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.99.62:35949
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab
--principal=kudu/127.13.99.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.62:35949
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.99.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18482
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786527905902985-13708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:49:56.343022 18482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:49:56.344089 18482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:49:56.353016 18493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:49:56.353173 18482 server_base.cc:1034] running on GCE node
W20260812 09:49:56.353058 18496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:49:56.355115 18492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:49:56.355983 18482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:49:56.357218 18482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:49:56.358420 18482 hybrid_clock.cc:648] HybridClock initialized: now 1786528196358410 us; error 31 us; skew 500 ppm
Aug 12 09:49:56 dist-test-slave-jq4t krb5kdc[18460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528196, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:49:56.361248 18482 init.cc:377] Logged in from keytab as kudu/127.13.99.62@KRBTEST.COM (short username kudu)
I20260812 09:49:56.362183 18482 webserver.cc:460] Webserver started at http://127.13.99.62:43251/ using document root <none> and password file <none>
I20260812 09:49:56.362793 18482 fs_manager.cc:359] Metadata directory not provided
I20260812 09:49:56.362891 18482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:49:56.363133 18482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:49:56.365272 18482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786527905902985-13708-0/minicluster-data/master-0/data/instance:
uuid: "04183f19977c4341b920604a256cca3d"
format_stamp: "Formatted at 2026-08-12 09:49:56 on dist-test-slave-jq4t"
server_key: "890dfaf853ba792f47c0e50dce36cf43"
server_key_iv: "b81735dc516a247a82b8d3ab046426ba"
server_key_version: "encryptionkey@0"
I20260812 09:49:56.365871 18482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786527905902985-13708-0/minicluster-data/master-0/wal/instance:
uuid: "04183f19977c4341b920604a256cca3d"
format_stamp: "Formatted at 2026-08-12 09:49:56 on dist-test-slave-jq4t"
server_key: "890dfaf853ba792f47c0e50dce36cf43"
server_key_iv: "b81735dc516a247a82b8d3ab046426ba"
server_key_version: "encryptionkey@0"
I20260812 09:49:56.369616 18482 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 09:49:56.371989 18506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:49:56.373220 18482 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 09:49:56.373374 18482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786527905902985-13708-0/minicluster-data/master-0/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786527905902985-13708-0/minicluster-data/master-0/wal
uuid: "04183f19977c4341b920604a256cca3d"
format_stamp: "Formatted at 2026-08-12 09:49:56 on dist-test-slave-jq4t"
server_key: "890dfaf853ba792f47c0e50dce36cf43"
server_key_iv: "b81735dc516a247a82b8d3ab046426ba"
server_key_version: "encryptionkey@0"
I20260812 09:49:56.373485 18482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786527905902985-13708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786527905902985-13708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786527905902985-13708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:49:56.396564 18482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:49:56.399626 18482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:49:56.399849 18482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:49:56.407279 18482 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.62:35949
I20260812 09:49:56.407307 18592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.62:35949 every 8 connection(s)
I20260812 09:49:56.408263 18482 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
I20260812 09:49:56.411060 18593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:49:56.413695 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 18482
I20260812 09:49:56.413805 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786527905902985-13708-0/minicluster-data/master-0/wal/instance
I20260812 09:49:56.414126 13708 external_mini_cluster.cc:1428] Setting key a327d0d2799053056deacf27e41ce569
I20260812 09:49:56.417214 18593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04183f19977c4341b920604a256cca3d: Bootstrap starting.
I20260812 09:49:56.419370 18593 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04183f19977c4341b920604a256cca3d: Neither blocks nor log segments found. Creating new log.
I20260812 09:49:56.420112 18593 log.cc:824] T 00000000000000000000000000000000 P 04183f19977c4341b920604a256cca3d: Log is configured to *not* fsync() on all Append() calls
I20260812 09:49:56.422107 18593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04183f19977c4341b920604a256cca3d: No bootstrap required, opened a new log
Aug 12 09:49:56 dist-test-slave-jq4t krb5kdc[18460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528196, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
I20260812 09:49:56.425611 18593 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04183f19977c4341b920604a256cca3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04183f19977c4341b920604a256cca3d" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 35949 } }
I20260812 09:49:56.425824 18593 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04183f19977c4341b920604a256cca3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:49:56.425885 18593 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04183f19977c4341b920604a256cca3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04183f19977c4341b920604a256cca3d, State: Initialized, Role: FOLLOWER
I20260812 09:49:56.426369 18593 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04183f19977c4341b920604a256cca3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04183f19977c4341b920604a256cca3d" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 35949 } }
I20260812 09:49:56.426498 18593 raft_consensus.cc:388] T 00000000000000000000000000000000 P 04183f19977c4341b920604a256cca3d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 09:49:56.426586 18593 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04183f19977c4341b920604a256cca3d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 09:49:56.426708 18593 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04183f19977c4341b920604a256cca3d [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:49:56.427557 18593 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04183f19977c4341b920604a256cca3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04183f19977c4341b920604a256cca3d" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 35949 } }
I20260812 09:49:56.427901 18593 leader_election.cc:302] T 00000000000000000000000000000000 P 04183f19977c4341b920604a256cca3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 04183f19977c4341b920604a256cca3d; no voters:
I20260812 09:49:56.428260 18593 leader_election.cc:288] T 00000000000000000000000000000000 P 04183f19977c4341b920604a256cca3d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 09:49:56.428326 18600 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04183f19977c4341b920604a256cca3d [term 1 FOLLOWER]: Leader election won for term 1
I20260812 09:49:56.428583 18600 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04183f19977c4341b920604a256cca3d [term 1 LEADER]: Becoming Leader. State: Replica: 04183f19977c4341b920604a256cca3d, State: Running, Role: LEADER
I20260812 09:49:56.428939 18600 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04183f19977c4341b920604a256cca3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04183f19977c4341b920604a256cca3d" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 35949 } }
I20260812 09:49:56.429423 18598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:56.415553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53817 (local address 127.13.99.62:35949)
0812 09:49:56.416016 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:56.416031 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:56.416066 (+ 35us) server_negotiation.cc:407] Connection header received
0812 09:49:56.416755 (+ 689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:56.416775 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:56.417113 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:56.417447 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:49:56.418245 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:56.419229 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:56.420137 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:56.420736 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:56.420932 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:56.423754 (+ 2822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:49:56.423759 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:49:56.423772 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:49:56.423799 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:49:56.426188 (+ 2389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:49:56.426663 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:49:56.426668 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:49:56.426673 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:49:56.426875 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:49:56.427187 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:49:56.427191 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:49:56.427193 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:49:56.427591 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:49:56.427797 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:56.428097 (+ 300us) server_negotiation.cc:299] Negotiation successful
0812 09:49:56.428452 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":131,"threads_started":1}
I20260812 09:49:56.429598 18593 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04183f19977c4341b920604a256cca3d [sys.catalog]: configured and running, proceeding with master startup.
I20260812 09:49:56.430683 18602 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04183f19977c4341b920604a256cca3d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "04183f19977c4341b920604a256cca3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04183f19977c4341b920604a256cca3d" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 35949 } } }
I20260812 09:49:56.430848 18602 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04183f19977c4341b920604a256cca3d [sys.catalog]: This master's current role is: LEADER
I20260812 09:49:56.431123 18601 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04183f19977c4341b920604a256cca3d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04183f19977c4341b920604a256cca3d. Latest consensus state: current_term: 1 leader_uuid: "04183f19977c4341b920604a256cca3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04183f19977c4341b920604a256cca3d" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 35949 } } }
I20260812 09:49:56.431213 18601 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04183f19977c4341b920604a256cca3d [sys.catalog]: This master's current role is: LEADER
I20260812 09:49:56.431255 18606 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 09:49:56.432399 13708 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 09:49:56.434572 18606 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 09:49:56.440389 18606 catalog_manager.cc:1347] Generated new cluster ID: df25f3fb40244076a4cbd640fcd018da
I20260812 09:49:56.440457 18606 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 09:49:56.446599 18598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:56.434552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33005 (local address 127.13.99.62:35949)
0812 09:49:56.434901 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:56.434905 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:56.437033 (+ 2128us) server_negotiation.cc:407] Connection header received
0812 09:49:56.437159 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:56.437162 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:56.437210 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:56.437296 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:49:56.438170 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:56.438888 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:56.440103 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:56.441222 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:56.441308 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:56.443136 (+ 1828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:49:56.443140 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:49:56.443143 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:49:56.443168 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:49:56.444882 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:49:56.445361 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:49:56.445365 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:49:56.445367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:49:56.445449 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:49:56.445749 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:49:56.445751 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:49:56.445753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:49:56.445949 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:49:56.446060 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:56.446332 (+ 272us) server_negotiation.cc:299] Negotiation successful
0812 09:49:56.446444 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281}
I20260812 09:49:56.458004 18606 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 09:49:56.458887 18606 catalog_manager.cc:1504] Loading token signing keys...
I20260812 09:49:56.464793 18606 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 04183f19977c4341b920604a256cca3d: Generated new TSK 0
I20260812 09:49:56.465490 18606 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 09:49:56.472157 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 18482
2026-08-12T09:49:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1661 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786527905902985-13708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 09:49:56 dist-test-slave-jq4t krb5kdc[18641](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 09:49:56 dist-test-slave-jq4t krb5kdc[18641](info): set up 2 sockets
Aug 12 09:49:56 dist-test-slave-jq4t krb5kdc[18641](info): commencing operation
krb5kdc: starting...
W20260812 09:49:57.761655 13708 mini_kdc.cc:121] Time spent starting KDC: real 1.261s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786527905902985-13708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786527905902985-13708-0/krb5kdc/test-user.keytab.
Aug 12 09:49:57 dist-test-slave-jq4t krb5kdc[18641](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528197, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T09:49:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T09:49:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.99.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.99.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.99.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab.
Entry for principal kudu/127.13.99.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab.
I20260812 09:49:57.891393 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786527905902985-13708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786527905902985-13708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786527905902985-13708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.62:38635
--webserver_interface=127.13.99.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab
--principal=kudu/127.13.99.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:33501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.99.62:38635
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:49:58.001909 18665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:49:58.002230 18665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:49:58.002311 18665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:49:58.006330 18665 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 09:49:58.006440 18665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:49:58.006479 18665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:49:58.006518 18665 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 09:49:58.006551 18665 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 09:49:58.012037 18665 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:33501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786527905902985-13708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786527905902985-13708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.99.62:38635
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab
--principal=kudu/127.13.99.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.62:38635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.99.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18665
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786527905902985-13708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:49:58.013239 18665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:49:58.014355 18665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 09:49:58.023437 18665 server_base.cc:1034] running on GCE node
W20260812 09:49:58.023339 18676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:49:58.023386 18679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:49:58.023339 18681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:49:58.024219 18665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:49:58.025385 18665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:49:58.026583 18665 hybrid_clock.cc:648] HybridClock initialized: now 1786528198026557 us; error 38 us; skew 500 ppm
Aug 12 09:49:58 dist-test-slave-jq4t krb5kdc[18641](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528198, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:49:58.029392 18665 init.cc:377] Logged in from keytab as kudu/127.13.99.62@KRBTEST.COM (short username kudu)
I20260812 09:49:58.030272 18665 webserver.cc:460] Webserver started at http://127.13.99.62:39447/ using document root <none> and password file <none>
I20260812 09:49:58.030823 18665 fs_manager.cc:359] Metadata directory not provided
I20260812 09:49:58.030884 18665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:49:58.031059 18665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:49:58.033136 18665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786527905902985-13708-0/minicluster-data/master-0/data/instance:
uuid: "d634ba0cf0b24b249ca6b4c9b96c950a"
format_stamp: "Formatted at 2026-08-12 09:49:58 on dist-test-slave-jq4t"
server_key: "9e93c246ded1ff43dd6163ffa09eb07b"
server_key_iv: "99b04c2383cf094fde52f393dcbc58b4"
server_key_version: "encryptionkey@0"
I20260812 09:49:58.033649 18665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786527905902985-13708-0/minicluster-data/master-0/wal/instance:
uuid: "d634ba0cf0b24b249ca6b4c9b96c950a"
format_stamp: "Formatted at 2026-08-12 09:49:58 on dist-test-slave-jq4t"
server_key: "9e93c246ded1ff43dd6163ffa09eb07b"
server_key_iv: "99b04c2383cf094fde52f393dcbc58b4"
server_key_version: "encryptionkey@0"
I20260812 09:49:58.037329 18665 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 09:49:58.039628 18696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:49:58.040949 18665 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260812 09:49:58.041092 18665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786527905902985-13708-0/minicluster-data/master-0/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786527905902985-13708-0/minicluster-data/master-0/wal
uuid: "d634ba0cf0b24b249ca6b4c9b96c950a"
format_stamp: "Formatted at 2026-08-12 09:49:58 on dist-test-slave-jq4t"
server_key: "9e93c246ded1ff43dd6163ffa09eb07b"
server_key_iv: "99b04c2383cf094fde52f393dcbc58b4"
server_key_version: "encryptionkey@0"
I20260812 09:49:58.041194 18665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786527905902985-13708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786527905902985-13708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786527905902985-13708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:49:58.063524 18665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:49:58.066509 18665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:49:58.066723 18665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:49:58.074164 18665 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.62:38635
I20260812 09:49:58.074178 18782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.62:38635 every 8 connection(s)
I20260812 09:49:58.075140 18665 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
I20260812 09:49:58.077543 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 18665
I20260812 09:49:58.077695 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786527905902985-13708-0/minicluster-data/master-0/wal/instance
I20260812 09:49:58.078023 13708 external_mini_cluster.cc:1428] Setting key b4b9e86cf4fbd569f74b49d58ab49a51
I20260812 09:49:58.078276 18784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:49:58.085804 18784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d634ba0cf0b24b249ca6b4c9b96c950a: Bootstrap starting.
I20260812 09:49:58.087391 18787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:58.079369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33325 (local address 127.13.99.62:38635)
0812 09:49:58.079788 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:58.079804 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:58.079840 (+ 36us) server_negotiation.cc:407] Connection header received
0812 09:49:58.080520 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:58.080553 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:58.080886 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:58.081166 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:49:58.081648 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:58.082737 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:58.083983 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:58.084540 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:58.084682 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:58.085617 (+ 935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:49:58.085622 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:49:58.085636 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 09:49:58.086089 (+ 453us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:49:58.086147 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 09:49:58.086172 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:49:58.086251 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:58.086423 (+ 172us) server_negotiation.cc:299] Negotiation successful
0812 09:49:58.086673 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":140,"threads_started":1}
I20260812 09:49:58.088292 18784 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d634ba0cf0b24b249ca6b4c9b96c950a: Neither blocks nor log segments found. Creating new log.
I20260812 09:49:58.089103 18784 log.cc:824] T 00000000000000000000000000000000 P d634ba0cf0b24b249ca6b4c9b96c950a: Log is configured to *not* fsync() on all Append() calls
I20260812 09:49:58.090248 13708 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 09:49:58.091423 18784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d634ba0cf0b24b249ca6b4c9b96c950a: No bootstrap required, opened a new log
I20260812 09:49:58.096208 18784 raft_consensus.cc:348] T 00000000000000000000000000000000 P d634ba0cf0b24b249ca6b4c9b96c950a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d634ba0cf0b24b249ca6b4c9b96c950a" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 38635 } }
I20260812 09:49:58.096539 18784 raft_consensus.cc:374] T 00000000000000000000000000000000 P d634ba0cf0b24b249ca6b4c9b96c950a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:49:58.096628 18784 raft_consensus.cc:729] T 00000000000000000000000000000000 P d634ba0cf0b24b249ca6b4c9b96c950a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d634ba0cf0b24b249ca6b4c9b96c950a, State: Initialized, Role: FOLLOWER
I20260812 09:49:58.096791 18787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:58.092595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34041 (local address 127.13.99.62:38635)
0812 09:49:58.092716 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:58.092720 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:58.092877 (+ 157us) server_negotiation.cc:407] Connection header received
0812 09:49:58.092999 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:58.093001 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:58.093042 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:58.093106 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:49:58.093399 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:58.093898 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:58.095079 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:49:58.095308 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:49:58.095382 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:49:58.096184 (+ 802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:49:58.096187 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:49:58.096189 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 09:49:58.096200 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:49:58.096224 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 09:49:58.096234 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:49:58.096293 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:58.096425 (+ 132us) server_negotiation.cc:299] Negotiation successful
0812 09:49:58.096469 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260812 09:49:58.097184 18784 consensus_queue.cc:260] T 00000000000000000000000000000000 P d634ba0cf0b24b249ca6b4c9b96c950a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d634ba0cf0b24b249ca6b4c9b96c950a" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 38635 } }
I20260812 09:49:58.097348 18784 raft_consensus.cc:388] T 00000000000000000000000000000000 P d634ba0cf0b24b249ca6b4c9b96c950a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 09:49:58.097419 18784 raft_consensus.cc:482] T 00000000000000000000000000000000 P d634ba0cf0b24b249ca6b4c9b96c950a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 09:49:58.097527 18784 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d634ba0cf0b24b249ca6b4c9b96c950a [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:49:58.098510 18784 raft_consensus.cc:504] T 00000000000000000000000000000000 P d634ba0cf0b24b249ca6b4c9b96c950a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d634ba0cf0b24b249ca6b4c9b96c950a" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 38635 } }
I20260812 09:49:58.098800 18784 leader_election.cc:302] T 00000000000000000000000000000000 P d634ba0cf0b24b249ca6b4c9b96c950a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d634ba0cf0b24b249ca6b4c9b96c950a; no voters:
I20260812 09:49:58.099093 18784 leader_election.cc:288] T 00000000000000000000000000000000 P d634ba0cf0b24b249ca6b4c9b96c950a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 09:49:58.099267 18797 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d634ba0cf0b24b249ca6b4c9b96c950a [term 1 FOLLOWER]: Leader election won for term 1
I20260812 09:49:58.099524 18797 raft_consensus.cc:686] T 00000000000000000000000000000000 P d634ba0cf0b24b249ca6b4c9b96c950a [term 1 LEADER]: Becoming Leader. State: Replica: d634ba0cf0b24b249ca6b4c9b96c950a, State: Running, Role: LEADER
I20260812 09:49:58.099865 18797 consensus_queue.cc:237] T 00000000000000000000000000000000 P d634ba0cf0b24b249ca6b4c9b96c950a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d634ba0cf0b24b249ca6b4c9b96c950a" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 38635 } }
I20260812 09:49:58.100455 18784 sys_catalog.cc:564] T 00000000000000000000000000000000 P d634ba0cf0b24b249ca6b4c9b96c950a [sys.catalog]: configured and running, proceeding with master startup.
I20260812 09:49:58.101603 18800 sys_catalog.cc:455] T 00000000000000000000000000000000 P d634ba0cf0b24b249ca6b4c9b96c950a [sys.catalog]: SysCatalogTable state changed. Reason: New leader d634ba0cf0b24b249ca6b4c9b96c950a. Latest consensus state: current_term: 1 leader_uuid: "d634ba0cf0b24b249ca6b4c9b96c950a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d634ba0cf0b24b249ca6b4c9b96c950a" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 38635 } } }
I20260812 09:49:58.101738 18800 sys_catalog.cc:458] T 00000000000000000000000000000000 P d634ba0cf0b24b249ca6b4c9b96c950a [sys.catalog]: This master's current role is: LEADER
I20260812 09:49:58.102088 18799 sys_catalog.cc:455] T 00000000000000000000000000000000 P d634ba0cf0b24b249ca6b4c9b96c950a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d634ba0cf0b24b249ca6b4c9b96c950a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d634ba0cf0b24b249ca6b4c9b96c950a" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 38635 } } }
I20260812 09:49:58.102178 18799 sys_catalog.cc:458] T 00000000000000000000000000000000 P d634ba0cf0b24b249ca6b4c9b96c950a [sys.catalog]: This master's current role is: LEADER
I20260812 09:49:58.102157 18817 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 09:49:58.105902 18817 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 09:49:58.111533 18817 catalog_manager.cc:1347] Generated new cluster ID: 900fe79a63934b389a0b9ebd9ba1d417
I20260812 09:49:58.111606 18817 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 09:49:58.125857 18817 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 09:49:58.126767 18817 catalog_manager.cc:1504] Loading token signing keys...
I20260812 09:49:58.137601 18817 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d634ba0cf0b24b249ca6b4c9b96c950a: Generated new TSK 0
I20260812 09:49:58.138311 18817 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 09:49:58.143959 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 18665
2026-08-12T09:49:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1672 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786527905902985-13708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 09:49:58 dist-test-slave-jq4t krb5kdc[18834](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 09:49:58 dist-test-slave-jq4t krb5kdc[18834](info): set up 2 sockets
Aug 12 09:49:58 dist-test-slave-jq4t krb5kdc[18834](info): commencing operation
krb5kdc: starting...
W20260812 09:49:59.422389 13708 mini_kdc.cc:121] Time spent starting KDC: real 1.249s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786527905902985-13708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786527905902985-13708-0/krb5kdc/test-user.keytab.
Aug 12 09:49:59 dist-test-slave-jq4t krb5kdc[18834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528199, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T09:49:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T09:49:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.99.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.99.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.99.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab.
Entry for principal kudu/127.13.99.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab.
I20260812 09:49:59.551898 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786527905902985-13708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786527905902985-13708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786527905902985-13708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.62:36997
--webserver_interface=127.13.99.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab
--principal=kudu/127.13.99.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:38421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.99.62:36997
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:49:59.666664 18854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:49:59.666993 18854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:49:59.667080 18854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:49:59.671115 18854 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 09:49:59.671208 18854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:49:59.671244 18854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:49:59.671284 18854 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 09:49:59.671317 18854 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 09:49:59.676508 18854 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:38421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786527905902985-13708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786527905902985-13708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.99.62:36997
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab
--principal=kudu/127.13.99.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.62:36997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.99.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18854
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786527905902985-13708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:49:59.677572 18854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:49:59.678614 18854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:49:59.687361 18861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:49:59.687371 18862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:49:59.687387 18867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:49:59.687645 18854 server_base.cc:1034] running on GCE node
I20260812 09:49:59.688375 18854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:49:59.689491 18854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:49:59.690696 18854 hybrid_clock.cc:648] HybridClock initialized: now 1786528199690683 us; error 25 us; skew 500 ppm
Aug 12 09:49:59 dist-test-slave-jq4t krb5kdc[18834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528199, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:49:59.693547 18854 init.cc:377] Logged in from keytab as kudu/127.13.99.62@KRBTEST.COM (short username kudu)
I20260812 09:49:59.694358 18854 webserver.cc:460] Webserver started at http://127.13.99.62:34079/ using document root <none> and password file <none>
I20260812 09:49:59.694883 18854 fs_manager.cc:359] Metadata directory not provided
I20260812 09:49:59.694941 18854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:49:59.695101 18854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:49:59.697170 18854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786527905902985-13708-0/minicluster-data/master-0/data/instance:
uuid: "581d4def282e497e9bffbe7ed0727f57"
format_stamp: "Formatted at 2026-08-12 09:49:59 on dist-test-slave-jq4t"
server_key: "f74fafd92433481811ddf87cf6d6782a"
server_key_iv: "2a5bb36b1d37583b6603c7deddbc4a9d"
server_key_version: "encryptionkey@0"
I20260812 09:49:59.697697 18854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786527905902985-13708-0/minicluster-data/master-0/wal/instance:
uuid: "581d4def282e497e9bffbe7ed0727f57"
format_stamp: "Formatted at 2026-08-12 09:49:59 on dist-test-slave-jq4t"
server_key: "f74fafd92433481811ddf87cf6d6782a"
server_key_iv: "2a5bb36b1d37583b6603c7deddbc4a9d"
server_key_version: "encryptionkey@0"
I20260812 09:49:59.702497 18854 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 09:49:59.705005 18883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:49:59.706377 18854 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 09:49:59.706494 18854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786527905902985-13708-0/minicluster-data/master-0/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786527905902985-13708-0/minicluster-data/master-0/wal
uuid: "581d4def282e497e9bffbe7ed0727f57"
format_stamp: "Formatted at 2026-08-12 09:49:59 on dist-test-slave-jq4t"
server_key: "f74fafd92433481811ddf87cf6d6782a"
server_key_iv: "2a5bb36b1d37583b6603c7deddbc4a9d"
server_key_version: "encryptionkey@0"
I20260812 09:49:59.706593 18854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786527905902985-13708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786527905902985-13708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786527905902985-13708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:49:59.711639 18854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:49:59.718837 18854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:49:59.719003 18854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:49:59.726692 18854 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.62:36997
I20260812 09:49:59.726697 18962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.62:36997 every 8 connection(s)
I20260812 09:49:59.727720 18854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
I20260812 09:49:59.729151 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 18854
I20260812 09:49:59.729305 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786527905902985-13708-0/minicluster-data/master-0/wal/instance
I20260812 09:49:59.729633 13708 external_mini_cluster.cc:1428] Setting key dd6585f30e1962323bf7d256dcfc5200
I20260812 09:49:59.731185 18964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:49:59.735070 18969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:59.731041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52241 (local address 127.13.99.62:36997)
0812 09:49:59.731525 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:59.731542 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:59.731582 (+ 40us) server_negotiation.cc:407] Connection header received
0812 09:49:59.732268 (+ 686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:59.732285 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:59.732658 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:59.732954 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:49:59.733789 (+ 835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:49:59.733794 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:49:59.733807 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 09:49:59.733817 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:49:59.733874 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 09:49:59.733894 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:49:59.734013 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:59.734127 (+ 114us) server_negotiation.cc:299] Negotiation successful
0812 09:49:59.734307 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":181,"threads_started":1}
I20260812 09:49:59.737680 18964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 581d4def282e497e9bffbe7ed0727f57: Bootstrap starting.
I20260812 09:49:59.738154 13708 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 09:49:59.740051 18964 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 581d4def282e497e9bffbe7ed0727f57: Neither blocks nor log segments found. Creating new log.
I20260812 09:49:59.741197 18964 log.cc:824] T 00000000000000000000000000000000 P 581d4def282e497e9bffbe7ed0727f57: Log is configured to *not* fsync() on all Append() calls
I20260812 09:49:59.741899 18969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:49:59.740206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32885 (local address 127.13.99.62:36997)
0812 09:49:59.740335 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0812 09:49:59.740340 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:49:59.740576 (+ 236us) server_negotiation.cc:407] Connection header received
0812 09:49:59.740709 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:49:59.740711 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:49:59.740740 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:49:59.740804 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:49:59.741498 (+ 694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:49:59.741501 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:49:59.741503 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 09:49:59.741514 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:49:59.741547 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 09:49:59.741557 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:49:59.741659 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0812 09:49:59.741742 (+ 83us) server_negotiation.cc:299] Negotiation successful
0812 09:49:59.741779 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260812 09:49:59.743309 18964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 581d4def282e497e9bffbe7ed0727f57: No bootstrap required, opened a new log
I20260812 09:49:59.746866 18964 raft_consensus.cc:348] T 00000000000000000000000000000000 P 581d4def282e497e9bffbe7ed0727f57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "581d4def282e497e9bffbe7ed0727f57" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 36997 } }
I20260812 09:49:59.747071 18964 raft_consensus.cc:374] T 00000000000000000000000000000000 P 581d4def282e497e9bffbe7ed0727f57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:49:59.747164 18964 raft_consensus.cc:729] T 00000000000000000000000000000000 P 581d4def282e497e9bffbe7ed0727f57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 581d4def282e497e9bffbe7ed0727f57, State: Initialized, Role: FOLLOWER
I20260812 09:49:59.747642 18964 consensus_queue.cc:260] T 00000000000000000000000000000000 P 581d4def282e497e9bffbe7ed0727f57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "581d4def282e497e9bffbe7ed0727f57" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 36997 } }
I20260812 09:49:59.747768 18964 raft_consensus.cc:388] T 00000000000000000000000000000000 P 581d4def282e497e9bffbe7ed0727f57 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 09:49:59.747834 18964 raft_consensus.cc:482] T 00000000000000000000000000000000 P 581d4def282e497e9bffbe7ed0727f57 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 09:49:59.747929 18964 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 581d4def282e497e9bffbe7ed0727f57 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:49:59.749032 18964 raft_consensus.cc:504] T 00000000000000000000000000000000 P 581d4def282e497e9bffbe7ed0727f57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "581d4def282e497e9bffbe7ed0727f57" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 36997 } }
I20260812 09:49:59.749367 18964 leader_election.cc:302] T 00000000000000000000000000000000 P 581d4def282e497e9bffbe7ed0727f57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 581d4def282e497e9bffbe7ed0727f57; no voters:
I20260812 09:49:59.749662 18964 leader_election.cc:288] T 00000000000000000000000000000000 P 581d4def282e497e9bffbe7ed0727f57 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 09:49:59.749795 18982 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 581d4def282e497e9bffbe7ed0727f57 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 09:49:59.750042 18982 raft_consensus.cc:686] T 00000000000000000000000000000000 P 581d4def282e497e9bffbe7ed0727f57 [term 1 LEADER]: Becoming Leader. State: Replica: 581d4def282e497e9bffbe7ed0727f57, State: Running, Role: LEADER
I20260812 09:49:59.750420 18982 consensus_queue.cc:237] T 00000000000000000000000000000000 P 581d4def282e497e9bffbe7ed0727f57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "581d4def282e497e9bffbe7ed0727f57" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 36997 } }
I20260812 09:49:59.750861 18964 sys_catalog.cc:564] T 00000000000000000000000000000000 P 581d4def282e497e9bffbe7ed0727f57 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 09:49:59.752202 18987 sys_catalog.cc:455] T 00000000000000000000000000000000 P 581d4def282e497e9bffbe7ed0727f57 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 581d4def282e497e9bffbe7ed0727f57. Latest consensus state: current_term: 1 leader_uuid: "581d4def282e497e9bffbe7ed0727f57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "581d4def282e497e9bffbe7ed0727f57" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 36997 } } }
I20260812 09:49:59.752256 18983 sys_catalog.cc:455] T 00000000000000000000000000000000 P 581d4def282e497e9bffbe7ed0727f57 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "581d4def282e497e9bffbe7ed0727f57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "581d4def282e497e9bffbe7ed0727f57" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 36997 } } }
I20260812 09:49:59.752323 18987 sys_catalog.cc:458] T 00000000000000000000000000000000 P 581d4def282e497e9bffbe7ed0727f57 [sys.catalog]: This master's current role is: LEADER
I20260812 09:49:59.752357 18983 sys_catalog.cc:458] T 00000000000000000000000000000000 P 581d4def282e497e9bffbe7ed0727f57 [sys.catalog]: This master's current role is: LEADER
I20260812 09:49:59.752755 18996 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 09:49:59.755411 18996 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 09:49:59.760892 18996 catalog_manager.cc:1347] Generated new cluster ID: c9fffa3a34ef45ea88794e6fa9b1ec49
I20260812 09:49:59.760951 18996 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 09:49:59.790948 18996 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 09:49:59.791286 18996 catalog_manager.cc:1504] Loading token signing keys...
I20260812 09:49:59.798482 18996 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 581d4def282e497e9bffbe7ed0727f57: Generated new TSK 0
I20260812 09:49:59.799320 18996 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 09:49:59.808207 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 18854
2026-08-12T09:49:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1670 ms)
[----------] 18 tests from AuthTokenIssuingTest (30150 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786527905902985-13708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 09:49:59 dist-test-slave-jq4t krb5kdc[19016](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 09:49:59 dist-test-slave-jq4t krb5kdc[19016](info): set up 2 sockets
Aug 12 09:49:59 dist-test-slave-jq4t krb5kdc[19016](info): commencing operation
krb5kdc: starting...
W20260812 09:50:01.090484 13708 mini_kdc.cc:121] Time spent starting KDC: real 1.248s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786527905902985-13708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786527905902985-13708-0/krb5kdc/test-user.keytab.
Aug 12 09:50:01 dist-test-slave-jq4t krb5kdc[19016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528201, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T09:50:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T09:50:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.99.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.99.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.99.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab.
Entry for principal kudu/127.13.99.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab.
I20260812 09:50:01.219430 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786527905902985-13708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786527905902985-13708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786527905902985-13708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.62:38981
--webserver_interface=127.13.99.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab
--principal=kudu/127.13.99.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:40791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.99.62:38981,127.13.99.61:38005,127.13.99.60:33401
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:50:01.330459 19043 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 09:50:01.330725 19043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:50:01.330749 19043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:50:01.330786 19043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:50:01.334720 19043 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 09:50:01.334770 19043 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:50:01.334789 19043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:50:01.334805 19043 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 09:50:01.334820 19043 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 09:50:01.339831 19043 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:40791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786527905902985-13708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786527905902985-13708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.99.62:38981,127.13.99.61:38005,127.13.99.60:33401
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab
--principal=kudu/127.13.99.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.62:38981
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.99.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19043
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786527905902985-13708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:50:01.340773 19043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:50:01.341701 19043 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:50:01.350157 19054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:50:01.350194 19051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:50:01.350368 19043 server_base.cc:1034] running on GCE node
W20260812 09:50:01.350157 19052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:50:01.351089 19043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:50:01.352177 19043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:50:01.353379 19043 hybrid_clock.cc:648] HybridClock initialized: now 1786528201353339 us; error 49 us; skew 500 ppm
Aug 12 09:50:01 dist-test-slave-jq4t krb5kdc[19016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528201, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:50:01.356158 19043 init.cc:377] Logged in from keytab as kudu/127.13.99.62@KRBTEST.COM (short username kudu)
I20260812 09:50:01.357034 19043 webserver.cc:460] Webserver started at http://127.13.99.62:42445/ using document root <none> and password file <none>
I20260812 09:50:01.357551 19043 fs_manager.cc:359] Metadata directory not provided
I20260812 09:50:01.357609 19043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:50:01.357766 19043 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:50:01.359727 19043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786527905902985-13708-0/minicluster-data/master-0/data/instance:
uuid: "688b60ed30ec4f4488cceb1a2671ddcc"
format_stamp: "Formatted at 2026-08-12 09:50:01 on dist-test-slave-jq4t"
server_key: "2d47bd54572b37895a7bd556b77bbf1a"
server_key_iv: "1166b27970ddf68d3adb69b47bf78b1b"
server_key_version: "encryptionkey@0"
I20260812 09:50:01.360271 19043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786527905902985-13708-0/minicluster-data/master-0/wal/instance:
uuid: "688b60ed30ec4f4488cceb1a2671ddcc"
format_stamp: "Formatted at 2026-08-12 09:50:01 on dist-test-slave-jq4t"
server_key: "2d47bd54572b37895a7bd556b77bbf1a"
server_key_iv: "1166b27970ddf68d3adb69b47bf78b1b"
server_key_version: "encryptionkey@0"
I20260812 09:50:01.363981 19043 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260812 09:50:01.366418 19062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:50:01.367671 19043 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 09:50:01.367831 19043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786527905902985-13708-0/minicluster-data/master-0/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786527905902985-13708-0/minicluster-data/master-0/wal
uuid: "688b60ed30ec4f4488cceb1a2671ddcc"
format_stamp: "Formatted at 2026-08-12 09:50:01 on dist-test-slave-jq4t"
server_key: "2d47bd54572b37895a7bd556b77bbf1a"
server_key_iv: "1166b27970ddf68d3adb69b47bf78b1b"
server_key_version: "encryptionkey@0"
I20260812 09:50:01.367928 19043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786527905902985-13708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786527905902985-13708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786527905902985-13708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:50:01.381709 19043 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:50:01.385430 19043 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:50:01.385632 19043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:50:01.392251 19043 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.62:38981
I20260812 09:50:01.392624 19145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.62:38981 every 8 connection(s)
I20260812 09:50:01.393253 19043 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
I20260812 09:50:01.395501 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 19043
I20260812 09:50:01.395642 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786527905902985-13708-0/minicluster-data/master-0/wal/instance
I20260812 09:50:01.395948 13708 external_mini_cluster.cc:1428] Setting key 076d977e7d011da37051ff7c9d519530
I20260812 09:50:01.396322 19146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:50:01.400388 19146 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.99.62" port: 38981 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.13.99.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.99.61@KRBTEST.COM" created.
Aug 12 09:50:01 dist-test-slave-jq4t krb5kdc[19016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528201, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.62@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
I20260812 09:50:01.417194 19150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:50:01.402758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:38981 (local address 127.0.0.1:35514)
0812 09:50:01.403535 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0812 09:50:01.403586 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0812 09:50:01.404776 (+ 1190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:50:01.405522 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:50:01.405527 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:50:01.405947 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:50:01.406214 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:50:01.406229 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:50:01.407515 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:50:01.407518 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:50:01.408581 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:50:01.408591 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:50:01.409530 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:50:01.409533 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:50:01.409798 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:50:01.410441 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:50:01.410467 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:50:01.412135 (+ 1668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:50:01.414176 (+ 2041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:50:01.414187 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:50:01.414196 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:50:01.414422 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:50:01.414752 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:50:01.414757 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:50:01.414759 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:50:01.414872 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:50:01.415460 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:50:01.415466 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:50:01.415607 (+ 141us) client_negotiation.cc:769] Sending connection context
0812 09:50:01.415796 (+ 189us) client_negotiation.cc:240] Negotiation successful
0812 09:50:01.416170 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"mutex_wait_us":180,"spinlock_wait_cycles":56704,"thread_start_us":153,"threads_started":1}
I20260812 09:50:01.417192 19149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:50:01.402758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35514 (local address 127.13.99.62:38981)
0812 09:50:01.403260 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0812 09:50:01.403280 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0812 09:50:01.403728 (+ 448us) server_negotiation.cc:407] Connection header received
0812 09:50:01.405109 (+ 1381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:50:01.405114 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:50:01.405308 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:50:01.405466 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:50:01.406384 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:50:01.407382 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:50:01.408704 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:50:01.409364 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:50:01.409611 (+ 247us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:50:01.412286 (+ 2675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:50:01.412292 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:50:01.412302 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:50:01.412322 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:50:01.414060 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:50:01.414547 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:50:01.414552 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:50:01.414553 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:50:01.414654 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:50:01.414965 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:50:01.414970 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:50:01.414972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:50:01.415334 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:50:01.415476 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0812 09:50:01.415874 (+ 398us) server_negotiation.cc:299] Negotiation successful
0812 09:50:01.416116 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":330,"thread_start_us":161,"threads_started":1}
I20260812 09:50:01.419642 19146 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.99.61" port: 38005 } has no permanent_uuid. Determining permanent_uuid...
I20260812 09:50:01.420365 19150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:50:01.420118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.61:38005 (local address 127.0.0.1:59458)
0812 09:50:01.420230 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0812 09:50:01.420307 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.61:38005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260812 09:50:01.420585 19063 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.99.61:38005: connect: Connection refused (error 111)
I20260812 09:50:01.421509 19150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:50:01.421294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.61:38005 (local address 127.0.0.1:59472)
0812 09:50:01.421403 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0812 09:50:01.421455 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.61:38005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260812 09:50:01.421681 19146 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.99.61:38005: Network error: Client connection negotiation failed: client connection to 127.13.99.61:38005: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.99.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.61.keytab.
Entry for principal kudu/127.13.99.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.61.keytab.
I20260812 09:50:01.423460 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786527905902985-13708-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786527905902985-13708-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786527905902985-13708-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786527905902985-13708-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.61:38005
--webserver_interface=127.13.99.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.61.keytab
--principal=kudu/127.13.99.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:40791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.99.62:38981,127.13.99.61:38005,127.13.99.60:33401
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 09:50:01.470896 19146 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.99.61" port: 38005 } attempt: 1
I20260812 09:50:01.471856 19150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:50:01.471555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.61:38005 (local address 127.0.0.1:59480)
0812 09:50:01.471713 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0812 09:50:01.471778 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.61:38005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260812 09:50:01.472672 19150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:50:01.472378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.61:38005 (local address 127.0.0.1:59488)
0812 09:50:01.472541 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0812 09:50:01.472597 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.61:38005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260812 09:50:01.472800 19146 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.99.61:38005: Network error: Client connection negotiation failed: client connection to 127.13.99.61:38005: connect: Connection refused (error 111)
W20260812 09:50:01.534773 19154 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 09:50:01.535102 19154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:50:01.535178 19154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:50:01.535261 19154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:50:01.539148 19154 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 09:50:01.539237 19154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:50:01.539275 19154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:50:01.539312 19154 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 09:50:01.539342 19154 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 09:50:01.540985 19146 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.99.61" port: 38005 } attempt: 2
I20260812 09:50:01.541903 19150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:50:01.541538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.61:38005 (local address 127.0.0.1:59498)
0812 09:50:01.541703 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0812 09:50:01.541773 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.61:38005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260812 09:50:01.542644 19150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:50:01.542414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.61:38005 (local address 127.0.0.1:59510)
0812 09:50:01.542533 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0812 09:50:01.542582 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.61:38005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260812 09:50:01.542758 19146 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.99.61:38005: Network error: Client connection negotiation failed: client connection to 127.13.99.61:38005: connect: Connection refused (error 111)
I20260812 09:50:01.544732 19154 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:40791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786527905902985-13708-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786527905902985-13708-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.13.99.62:38981,127.13.99.61:38005,127.13.99.60:33401
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.61.keytab
--principal=kudu/127.13.99.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.61:38005
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786527905902985-13708-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.13.99.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19154
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786527905902985-13708-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:50:01.545781 19154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:50:01.546789 19154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:50:01.555264 19161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:50:01.555264 19160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:50:01.555271 19164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:50:01.555661 19154 server_base.cc:1034] running on GCE node
I20260812 09:50:01.556293 19154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:50:01.557475 19154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:50:01.558679 19154 hybrid_clock.cc:648] HybridClock initialized: now 1786528201558664 us; error 27 us; skew 500 ppm
Aug 12 09:50:01 dist-test-slave-jq4t krb5kdc[19016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528201, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:50:01.561600 19154 init.cc:377] Logged in from keytab as kudu/127.13.99.61@KRBTEST.COM (short username kudu)
I20260812 09:50:01.562429 19154 webserver.cc:460] Webserver started at http://127.13.99.61:38627/ using document root <none> and password file <none>
I20260812 09:50:01.562958 19154 fs_manager.cc:359] Metadata directory not provided
I20260812 09:50:01.563015 19154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:50:01.563186 19154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:50:01.565301 19154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786527905902985-13708-0/minicluster-data/master-1/data/instance:
uuid: "1da86a8a31214e70aac5ada3147c94ed"
format_stamp: "Formatted at 2026-08-12 09:50:01 on dist-test-slave-jq4t"
server_key: "6c1eac75c1e5617cfd49595f97353c24"
server_key_iv: "615a0812d51fb2e3dc55e357adbf7e99"
server_key_version: "encryptionkey@0"
I20260812 09:50:01.565825 19154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786527905902985-13708-0/minicluster-data/master-1/wal/instance:
uuid: "1da86a8a31214e70aac5ada3147c94ed"
format_stamp: "Formatted at 2026-08-12 09:50:01 on dist-test-slave-jq4t"
server_key: "6c1eac75c1e5617cfd49595f97353c24"
server_key_iv: "615a0812d51fb2e3dc55e357adbf7e99"
server_key_version: "encryptionkey@0"
I20260812 09:50:01.569523 19154 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260812 09:50:01.572027 19175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:50:01.573359 19154 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 09:50:01.573477 19154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786527905902985-13708-0/minicluster-data/master-1/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786527905902985-13708-0/minicluster-data/master-1/wal
uuid: "1da86a8a31214e70aac5ada3147c94ed"
format_stamp: "Formatted at 2026-08-12 09:50:01 on dist-test-slave-jq4t"
server_key: "6c1eac75c1e5617cfd49595f97353c24"
server_key_iv: "615a0812d51fb2e3dc55e357adbf7e99"
server_key_version: "encryptionkey@0"
I20260812 09:50:01.573565 19154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786527905902985-13708-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786527905902985-13708-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786527905902985-13708-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:50:01.593084 19154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:50:01.596032 19154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:50:01.596190 19154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:50:01.603564 19154 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.61:38005
I20260812 09:50:01.603888 19265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.61:38005 every 8 connection(s)
I20260812 09:50:01.604607 19154 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786527905902985-13708-0/minicluster-data/master-1/data/info.pb
I20260812 09:50:01.607465 19266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:50:01.609972 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 19154
I20260812 09:50:01.610080 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786527905902985-13708-0/minicluster-data/master-1/wal/instance
I20260812 09:50:01.610364 13708 external_mini_cluster.cc:1428] Setting key 4634865febcf4b56d7637375bd1f160e
I20260812 09:50:01.611482 19266 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.99.62" port: 38981 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.13.99.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.99.60@KRBTEST.COM" created.
Aug 12 09:50:01 dist-test-slave-jq4t krb5kdc[19016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528201, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.61@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
I20260812 09:50:01.627887 19149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:50:01.613676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35524 (local address 127.13.99.62:38981)
0812 09:50:01.613861 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0812 09:50:01.613867 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 09:50:01.614678 (+ 811us) server_negotiation.cc:407] Connection header received
0812 09:50:01.615623 (+ 945us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:50:01.615630 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:50:01.615703 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:50:01.615815 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:50:01.617089 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:50:01.618126 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:50:01.619447 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:50:01.619874 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:50:01.620008 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:50:01.623658 (+ 3650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:50:01.623662 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:50:01.623664 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:50:01.623690 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:50:01.625474 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:50:01.626014 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:50:01.626025 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:50:01.626027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:50:01.626111 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:50:01.626533 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:50:01.626540 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:50:01.626544 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:50:01.626813 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:50:01.626923 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0812 09:50:01.627479 (+ 556us) server_negotiation.cc:299] Negotiation successful
0812 09:50:01.627658 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260812 09:50:01.628676 19272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:50:01.614019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:38981 (local address 127.0.0.1:35524)
0812 09:50:01.614499 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0812 09:50:01.614553 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0812 09:50:01.615400 (+ 847us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:50:01.615999 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:50:01.616007 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:50:01.616353 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:50:01.616913 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:50:01.616937 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:50:01.618297 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:50:01.618300 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:50:01.619296 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:50:01.619308 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:50:01.620044 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:50:01.620048 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:50:01.620206 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:50:01.620920 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:50:01.620938 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:50:01.623477 (+ 2539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:50:01.625605 (+ 2128us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:50:01.625610 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:50:01.625623 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:50:01.625906 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:50:01.626251 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:50:01.626254 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:50:01.626255 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:50:01.626415 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:50:01.626925 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:50:01.626931 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:50:01.627197 (+ 266us) client_negotiation.cc:769] Sending connection context
0812 09:50:01.627361 (+ 164us) client_negotiation.cc:240] Negotiation successful
0812 09:50:01.627632 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":149,"threads_started":1}
I20260812 09:50:01.630226 19266 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.99.61" port: 38005 } has no permanent_uuid. Determining permanent_uuid...
I20260812 09:50:01.633916 19146 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.99.61" port: 38005 } attempt: 3
Aug 12 09:50:01 dist-test-slave-jq4t krb5kdc[19016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528201, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.61@KRBTEST.COM for kudu/127.13.99.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.99.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.60.keytab.
Entry for principal kudu/127.13.99.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.60.keytab.
Aug 12 09:50:01 dist-test-slave-jq4t krb5kdc[19016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528201, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.62@KRBTEST.COM for kudu/127.13.99.61@KRBTEST.COM
I20260812 09:50:01.640093 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786527905902985-13708-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786527905902985-13708-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786527905902985-13708-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786527905902985-13708-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.60:33401
--webserver_interface=127.13.99.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.60.keytab
--principal=kudu/127.13.99.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:40791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.99.62:38981,127.13.99.61:38005,127.13.99.60:33401
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 09:50:01.640650 19272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:50:01.630783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.61:38005 (local address 127.0.0.1:59524)
0812 09:50:01.630920 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0812 09:50:01.630936 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 09:50:01.631012 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:50:01.631601 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:50:01.631604 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:50:01.631779 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:50:01.631857 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:50:01.631862 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:50:01.632885 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:50:01.632892 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:50:01.633424 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:50:01.633433 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:50:01.634203 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:50:01.634207 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:50:01.634319 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:50:01.634864 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:50:01.634882 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:50:01.636556 (+ 1674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:50:01.639042 (+ 2486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:50:01.639047 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:50:01.639052 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:50:01.639257 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:50:01.639567 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:50:01.639571 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:50:01.639573 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:50:01.639657 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:50:01.640103 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:50:01.640106 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:50:01.640198 (+ 92us) client_negotiation.cc:769] Sending connection context
0812 09:50:01.640316 (+ 118us) client_negotiation.cc:240] Negotiation successful
0812 09:50:01.640434 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260812 09:50:01.641698 19277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:50:01.630941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59524 (local address 127.13.99.61:38005)
0812 09:50:01.631256 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0812 09:50:01.631265 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 09:50:01.631284 (+ 19us) server_negotiation.cc:407] Connection header received
0812 09:50:01.631316 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:50:01.631320 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:50:01.631462 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:50:01.631611 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:50:01.631967 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:50:01.632771 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:50:01.633554 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:50:01.633986 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:50:01.634093 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:50:01.636770 (+ 2677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:50:01.636779 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:50:01.636791 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:50:01.636812 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:50:01.638915 (+ 2103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:50:01.639369 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:50:01.639375 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:50:01.639377 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:50:01.639455 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:50:01.639748 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:50:01.639753 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:50:01.639755 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:50:01.639987 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:50:01.640132 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0812 09:50:01.640649 (+ 517us) server_negotiation.cc:299] Negotiation successful
0812 09:50:01.640989 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":130,"threads_started":1}
I20260812 09:50:01.644212 19266 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.99.60" port: 33401 } has no permanent_uuid. Determining permanent_uuid...
I20260812 09:50:01.644840 19278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:50:01.634439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59534 (local address 127.13.99.61:38005)
0812 09:50:01.634733 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0812 09:50:01.634740 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 09:50:01.634946 (+ 206us) server_negotiation.cc:407] Connection header received
0812 09:50:01.635066 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:50:01.635068 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:50:01.635103 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:50:01.635172 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:50:01.635663 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:50:01.636551 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:50:01.637545 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:50:01.638210 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:50:01.638323 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:50:01.640723 (+ 2400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:50:01.640729 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:50:01.640735 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:50:01.640767 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:50:01.642769 (+ 2002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:50:01.643347 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:50:01.643350 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:50:01.643351 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:50:01.643425 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:50:01.643731 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:50:01.643737 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:50:01.643742 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:50:01.643994 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:50:01.644102 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0812 09:50:01.644543 (+ 441us) server_negotiation.cc:299] Negotiation successful
0812 09:50:01.644663 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":114,"threads_started":1}
I20260812 09:50:01.645139 19150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:50:01.634368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.61:38005 (local address 127.0.0.1:59534)
0812 09:50:01.634864 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0812 09:50:01.634881 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 09:50:01.634980 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:50:01.635218 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:50:01.635221 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:50:01.635406 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:50:01.635544 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:50:01.635550 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:50:01.636670 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:50:01.636677 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:50:01.637409 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:50:01.637429 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:50:01.638322 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:50:01.638325 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:50:01.638449 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:50:01.638938 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:50:01.638957 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:50:01.640545 (+ 1588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:50:01.642896 (+ 2351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:50:01.642900 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:50:01.642902 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:50:01.643208 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:50:01.643504 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:50:01.643507 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:50:01.643510 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:50:01.643615 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:50:01.644204 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:50:01.644207 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:50:01.644339 (+ 132us) client_negotiation.cc:769] Sending connection context
0812 09:50:01.644457 (+ 118us) client_negotiation.cc:240] Negotiation successful
0812 09:50:01.644991 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260812 09:50:01.645675 19272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:50:01.644734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.60:33401 (local address 127.0.0.1:46764)
0812 09:50:01.645351 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0812 09:50:01.645434 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.60:33401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":539}
W20260812 09:50:01.645848 19176 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.99.60:33401: connect: Connection refused (error 111)
I20260812 09:50:01.645843 19146 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.99.60" port: 33401 } has no permanent_uuid. Determining permanent_uuid...
I20260812 09:50:01.646589 19150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:50:01.646277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.60:33401 (local address 127.0.0.1:46780)
0812 09:50:01.646464 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0812 09:50:01.646517 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.60:33401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260812 09:50:01.647209 19150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:50:01.646962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.60:33401 (local address 127.0.0.1:46796)
0812 09:50:01.647102 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0812 09:50:01.647152 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.60:33401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260812 09:50:01.647218 19272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:50:01.646569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.60:33401 (local address 127.0.0.1:46782)
0812 09:50:01.647123 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0812 09:50:01.647169 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.60:33401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227}
W20260812 09:50:01.647354 19146 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.99.60:33401: Network error: Client connection negotiation failed: client connection to 127.13.99.60:33401: connect: Connection refused (error 111)
W20260812 09:50:01.647481 19266 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.99.60:33401: Network error: Client connection negotiation failed: client connection to 127.13.99.60:33401: connect: Connection refused (error 111)
I20260812 09:50:01.678503 19146 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.99.60" port: 33401 } attempt: 1
I20260812 09:50:01.679381 19150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:50:01.679029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.60:33401 (local address 127.0.0.1:46800)
0812 09:50:01.679205 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0812 09:50:01.679286 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.60:33401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260812 09:50:01.680131 19150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:50:01.679884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.60:33401 (local address 127.0.0.1:46810)
0812 09:50:01.680008 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0812 09:50:01.680065 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.60:33401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260812 09:50:01.680279 19146 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.99.60:33401: Network error: Client connection negotiation failed: client connection to 127.13.99.60:33401: connect: Connection refused (error 111)
I20260812 09:50:01.696797 19266 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.99.60" port: 33401 } attempt: 1
I20260812 09:50:01.697706 19272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:50:01.697389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.60:33401 (local address 127.0.0.1:46820)
0812 09:50:01.697560 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0812 09:50:01.697628 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.60:33401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260812 09:50:01.698513 19272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:50:01.698295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.60:33401 (local address 127.0.0.1:46828)
0812 09:50:01.698403 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0812 09:50:01.698451 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.60:33401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260812 09:50:01.698724 19266 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.99.60:33401: Network error: Client connection negotiation failed: client connection to 127.13.99.60:33401: connect: Connection refused (error 111)
I20260812 09:50:01.755456 19146 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.99.60" port: 33401 } attempt: 2
I20260812 09:50:01.756321 19150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:50:01.755986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.60:33401 (local address 127.0.0.1:46832)
0812 09:50:01.756155 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0812 09:50:01.756225 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.60:33401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260812 09:50:01.757644 19150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:50:01.757401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.60:33401 (local address 127.0.0.1:46836)
0812 09:50:01.757514 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0812 09:50:01.757577 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.60:33401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260812 09:50:01.757783 19146 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.99.60:33401: Network error: Client connection negotiation failed: client connection to 127.13.99.60:33401: connect: Connection refused (error 111)
W20260812 09:50:01.759243 19279 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 09:50:01.759505 19279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:50:01.759550 19279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:50:01.759636 19279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:50:01.763609 19279 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 09:50:01.763659 19279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:50:01.763698 19279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:50:01.763732 19279 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 09:50:01.763767 19279 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 09:50:01.766898 19266 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.99.60" port: 33401 } attempt: 2
I20260812 09:50:01.767585 19272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:50:01.767339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.60:33401 (local address 127.0.0.1:46838)
0812 09:50:01.767465 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0812 09:50:01.767518 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.60:33401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260812 09:50:01.768258 19272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:50:01.768026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.60:33401 (local address 127.0.0.1:46840)
0812 09:50:01.768152 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0812 09:50:01.768206 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.60:33401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260812 09:50:01.768383 19266 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.99.60:33401: Network error: Client connection negotiation failed: client connection to 127.13.99.60:33401: connect: Connection refused (error 111)
I20260812 09:50:01.769213 19279 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:40791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786527905902985-13708-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786527905902985-13708-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.13.99.62:38981,127.13.99.61:38005,127.13.99.60:33401
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.60.keytab
--principal=kudu/127.13.99.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.60:33401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786527905902985-13708-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.13.99.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19279
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786527905902985-13708-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:50:01.770164 19279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:50:01.771185 19279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:50:01.779422 19296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:50:01.779461 19294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:50:01.779469 19298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:50:01.779618 19279 server_base.cc:1034] running on GCE node
I20260812 09:50:01.780390 19279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:50:01.781461 19279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:50:01.782676 19279 hybrid_clock.cc:648] HybridClock initialized: now 1786528201782658 us; error 39 us; skew 500 ppm
Aug 12 09:50:01 dist-test-slave-jq4t krb5kdc[19016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528201, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:50:01.785497 19279 init.cc:377] Logged in from keytab as kudu/127.13.99.60@KRBTEST.COM (short username kudu)
I20260812 09:50:01.786368 19279 webserver.cc:460] Webserver started at http://127.13.99.60:36653/ using document root <none> and password file <none>
I20260812 09:50:01.786897 19279 fs_manager.cc:359] Metadata directory not provided
I20260812 09:50:01.786958 19279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:50:01.787119 19279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:50:01.789208 19279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786527905902985-13708-0/minicluster-data/master-2/data/instance:
uuid: "c18c1248371649e3945385c9eebc937e"
format_stamp: "Formatted at 2026-08-12 09:50:01 on dist-test-slave-jq4t"
server_key: "0f6f2ab0a86a5132381d2984f487c645"
server_key_iv: "bd8104f98772dff40fde62578326181d"
server_key_version: "encryptionkey@0"
I20260812 09:50:01.789709 19279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786527905902985-13708-0/minicluster-data/master-2/wal/instance:
uuid: "c18c1248371649e3945385c9eebc937e"
format_stamp: "Formatted at 2026-08-12 09:50:01 on dist-test-slave-jq4t"
server_key: "0f6f2ab0a86a5132381d2984f487c645"
server_key_iv: "bd8104f98772dff40fde62578326181d"
server_key_version: "encryptionkey@0"
I20260812 09:50:01.793412 19279 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 09:50:01.795795 19307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:50:01.797412 19279 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 09:50:01.797539 19279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786527905902985-13708-0/minicluster-data/master-2/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786527905902985-13708-0/minicluster-data/master-2/wal
uuid: "c18c1248371649e3945385c9eebc937e"
format_stamp: "Formatted at 2026-08-12 09:50:01 on dist-test-slave-jq4t"
server_key: "0f6f2ab0a86a5132381d2984f487c645"
server_key_iv: "bd8104f98772dff40fde62578326181d"
server_key_version: "encryptionkey@0"
I20260812 09:50:01.797680 19279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786527905902985-13708-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786527905902985-13708-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786527905902985-13708-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:50:01.815481 19279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:50:01.818559 19279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:50:01.818712 19279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:50:01.825682 19279 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.60:33401
I20260812 09:50:01.825971 19391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.60:33401 every 8 connection(s)
I20260812 09:50:01.826695 19279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786527905902985-13708-0/minicluster-data/master-2/data/info.pb
I20260812 09:50:01.828244 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 19279
I20260812 09:50:01.828384 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786527905902985-13708-0/minicluster-data/master-2/wal/instance
I20260812 09:50:01.828680 13708 external_mini_cluster.cc:1428] Setting key 2545009a82407b18123703aedeadec6f
I20260812 09:50:01.829746 19393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:50:01.834553 19393 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.99.62" port: 38981 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 09:50:01 dist-test-slave-jq4t krb5kdc[19016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528201, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.99.60@KRBTEST.COM
Aug 12 09:50:01 dist-test-slave-jq4t krb5kdc[19016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528201, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.60@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
I20260812 09:50:01.844636 19396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:50:01.830032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46856 (local address 127.13.99.60:33401)
0812 09:50:01.830441 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0812 09:50:01.830460 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0812 09:50:01.830501 (+ 41us) server_negotiation.cc:407] Connection header received
0812 09:50:01.831370 (+ 869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:50:01.831392 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:50:01.831786 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:50:01.832144 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:50:01.832901 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:50:01.834522 (+ 1621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:50:01.835506 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:50:01.836160 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:50:01.836349 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:50:01.839041 (+ 2692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:50:01.839047 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:50:01.839056 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:50:01.839084 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:50:01.841153 (+ 2069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:50:01.841754 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:50:01.841764 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:50:01.841773 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:50:01.841921 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:50:01.842202 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:50:01.842205 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:50:01.842207 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:50:01.842662 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:50:01.842881 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0812 09:50:01.843206 (+ 325us) server_negotiation.cc:299] Negotiation successful
0812 09:50:01.843515 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":105,"threads_started":1}
I20260812 09:50:01.846367 19149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:50:01.836413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35530 (local address 127.13.99.62:38981)
0812 09:50:01.836586 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0812 09:50:01.836593 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 09:50:01.836996 (+ 403us) server_negotiation.cc:407] Connection header received
0812 09:50:01.837179 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:50:01.837182 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:50:01.837234 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:50:01.837323 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:50:01.838030 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:50:01.838653 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:50:01.839774 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:50:01.840210 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:50:01.840345 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:50:01.842938 (+ 2593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:50:01.842942 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:50:01.842945 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:50:01.842981 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:50:01.844594 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:50:01.845097 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:50:01.845102 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:50:01.845105 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:50:01.845199 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:50:01.845464 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:50:01.845467 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:50:01.845469 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:50:01.845687 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:50:01.845790 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0812 09:50:01.846151 (+ 361us) server_negotiation.cc:299] Negotiation successful
0812 09:50:01.846254 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260812 09:50:01.846434 19397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:50:01.836363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:38981 (local address 127.0.0.1:35530)
0812 09:50:01.836840 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0812 09:50:01.836905 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0812 09:50:01.837036 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:50:01.837379 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:50:01.837385 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:50:01.837730 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:50:01.837860 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:50:01.837866 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:50:01.838789 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:50:01.838795 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:50:01.839643 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:50:01.839654 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:50:01.840349 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:50:01.840352 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:50:01.840443 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:50:01.840967 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:50:01.840979 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:50:01.842750 (+ 1771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:50:01.844714 (+ 1964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:50:01.844725 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:50:01.844731 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:50:01.844991 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:50:01.845293 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:50:01.845295 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:50:01.845296 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:50:01.845370 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:50:01.845790 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:50:01.845795 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:50:01.845932 (+ 137us) client_negotiation.cc:769] Sending connection context
0812 09:50:01.846077 (+ 145us) client_negotiation.cc:240] Negotiation successful
0812 09:50:01.846248 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":194,"threads_started":1}
I20260812 09:50:01.847736 19393 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.99.61" port: 38005 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 09:50:01 dist-test-slave-jq4t krb5kdc[19016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528201, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
Aug 12 09:50:01 dist-test-slave-jq4t krb5kdc[19016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528201, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.60@KRBTEST.COM for kudu/127.13.99.61@KRBTEST.COM
I20260812 09:50:01.856032 19397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:50:01.848250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.61:38005 (local address 127.0.0.1:59542)
0812 09:50:01.848400 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0812 09:50:01.848413 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 09:50:01.848515 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:50:01.848765 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:50:01.848767 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:50:01.848892 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:50:01.848998 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:50:01.849003 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:50:01.849786 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:50:01.849790 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:50:01.850491 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:50:01.850503 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:50:01.851207 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:50:01.851212 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:50:01.851348 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:50:01.851738 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:50:01.851758 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:50:01.853010 (+ 1252us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:50:01.854591 (+ 1581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:50:01.854594 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:50:01.854595 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:50:01.854774 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:50:01.855073 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:50:01.855078 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:50:01.855080 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:50:01.855159 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:50:01.855542 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:50:01.855545 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:50:01.855637 (+ 92us) client_negotiation.cc:769] Sending connection context
0812 09:50:01.855741 (+ 104us) client_negotiation.cc:240] Negotiation successful
0812 09:50:01.855862 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89}
I20260812 09:50:01.856098 19149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:50:01.847371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35534 (local address 127.13.99.62:38981)
0812 09:50:01.847616 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0812 09:50:01.847620 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:50:01.847633 (+ 13us) server_negotiation.cc:407] Connection header received
0812 09:50:01.847667 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:50:01.847670 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:50:01.847710 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:50:01.847796 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:50:01.848561 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:50:01.849195 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:50:01.850015 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:50:01.850216 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:50:01.850291 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:50:01.852812 (+ 2521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:50:01.852816 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:50:01.852818 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:50:01.852850 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:50:01.854440 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:50:01.854869 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:50:01.854873 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:50:01.854875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:50:01.854965 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:50:01.855248 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:50:01.855252 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:50:01.855254 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:50:01.855446 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:50:01.855670 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0812 09:50:01.855799 (+ 129us) server_negotiation.cc:299] Negotiation successful
0812 09:50:01.855912 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158}
I20260812 09:50:01.856205 19278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:50:01.848350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59542 (local address 127.13.99.61:38005)
0812 09:50:01.848509 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0812 09:50:01.848516 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 09:50:01.848535 (+ 19us) server_negotiation.cc:407] Connection header received
0812 09:50:01.848604 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:50:01.848607 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:50:01.848660 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:50:01.848743 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:50:01.849115 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:50:01.849676 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:50:01.850831 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:50:01.851077 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:50:01.851161 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:50:01.853153 (+ 1992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:50:01.853156 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:50:01.853158 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:50:01.853180 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:50:01.854469 (+ 1289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:50:01.854884 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:50:01.854888 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:50:01.854890 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:50:01.854980 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:50:01.855252 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:50:01.855256 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:50:01.855258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:50:01.855438 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:50:01.855552 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0812 09:50:01.855794 (+ 242us) server_negotiation.cc:299] Negotiation successful
0812 09:50:01.855915 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260812 09:50:01.856871 19393 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.99.60" port: 33401 } has no permanent_uuid. Determining permanent_uuid...
I20260812 09:50:01.856957 19146 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.99.60" port: 33401 } attempt: 3
I20260812 09:50:01.859534 19266 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.99.60" port: 33401 } attempt: 3
Aug 12 09:50:01 dist-test-slave-jq4t krb5kdc[19016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528201, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.60@KRBTEST.COM for kudu/127.13.99.60@KRBTEST.COM
Aug 12 09:50:01 dist-test-slave-jq4t krb5kdc[19016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528201, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.62@KRBTEST.COM for kudu/127.13.99.60@KRBTEST.COM
Aug 12 09:50:01 dist-test-slave-jq4t krb5kdc[19016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528201, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.99.61@KRBTEST.COM
Aug 12 09:50:01 dist-test-slave-jq4t krb5kdc[19016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528201, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.61@KRBTEST.COM for kudu/127.13.99.60@KRBTEST.COM
I20260812 09:50:01.867002 19396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:50:01.857564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46874 (local address 127.13.99.60:33401)
0812 09:50:01.857677 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0812 09:50:01.857684 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 09:50:01.857715 (+ 31us) server_negotiation.cc:407] Connection header received
0812 09:50:01.857846 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:50:01.857852 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:50:01.857887 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:50:01.857955 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:50:01.858477 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:50:01.859251 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:50:01.860431 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:50:01.860792 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:50:01.860858 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:50:01.863074 (+ 2216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:50:01.863077 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:50:01.863080 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:50:01.863101 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:50:01.865144 (+ 2043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:50:01.865643 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:50:01.865649 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:50:01.865653 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:50:01.865734 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:50:01.866044 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:50:01.866047 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:50:01.866049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:50:01.866256 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:50:01.866353 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0812 09:50:01.866689 (+ 336us) server_negotiation.cc:299] Negotiation successful
0812 09:50:01.866820 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260812 09:50:01.867427 19397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:50:01.857327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.60:33401 (local address 127.0.0.1:46874)
0812 09:50:01.857660 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0812 09:50:01.857671 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 09:50:01.857753 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:50:01.858075 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:50:01.858078 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:50:01.858249 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:50:01.858344 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:50:01.858350 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:50:01.859365 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:50:01.859368 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:50:01.860323 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:50:01.860333 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:50:01.860986 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:50:01.860992 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:50:01.861073 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:50:01.861487 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:50:01.861505 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:50:01.862941 (+ 1436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:50:01.865277 (+ 2336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:50:01.865282 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:50:01.865287 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:50:01.865531 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:50:01.865831 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:50:01.865836 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:50:01.865840 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:50:01.865919 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:50:01.866393 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:50:01.866396 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:50:01.866496 (+ 100us) client_negotiation.cc:769] Sending connection context
0812 09:50:01.867139 (+ 643us) client_negotiation.cc:240] Negotiation successful
0812 09:50:01.867252 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"mutex_wait_us":4}
I20260812 09:50:01.868708 19278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:50:01.857491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59552 (local address 127.13.99.61:38005)
0812 09:50:01.857604 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0812 09:50:01.857609 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:50:01.857622 (+ 13us) server_negotiation.cc:407] Connection header received
0812 09:50:01.857655 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:50:01.857658 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:50:01.857702 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:50:01.857781 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:50:01.858405 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:50:01.858898 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:50:01.859632 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:50:01.859953 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:50:01.860066 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:50:01.864423 (+ 4357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:50:01.864426 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:50:01.864429 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:50:01.864456 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:50:01.866187 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:50:01.866842 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:50:01.866846 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:50:01.866850 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:50:01.866939 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:50:01.867361 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:50:01.867365 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:50:01.867367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:50:01.867569 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:50:01.867744 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0812 09:50:01.868375 (+ 631us) server_negotiation.cc:299] Negotiation successful
0812 09:50:01.868537 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260812 09:50:01.868973 19150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:50:01.857360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.60:33401 (local address 127.0.0.1:46864)
0812 09:50:01.857534 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0812 09:50:01.857554 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 09:50:01.857617 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:50:01.858330 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:50:01.858335 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:50:01.858511 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:50:01.858600 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:50:01.858607 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:50:01.860098 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:50:01.860101 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:50:01.860899 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:50:01.860910 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:50:01.861536 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:50:01.861542 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:50:01.861640 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:50:01.862046 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:50:01.862066 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:50:01.864220 (+ 2154us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:50:01.867266 (+ 3046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:50:01.867270 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:50:01.867275 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:50:01.867534 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:50:01.867883 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:50:01.867889 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:50:01.867891 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:50:01.867990 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:50:01.868451 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:50:01.868467 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:50:01.868594 (+ 127us) client_negotiation.cc:769] Sending connection context
0812 09:50:01.868683 (+ 89us) client_negotiation.cc:240] Negotiation successful
0812 09:50:01.868822 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100}
I20260812 09:50:01.869508 19398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:50:01.857755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46864 (local address 127.13.99.60:33401)
0812 09:50:01.858102 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0812 09:50:01.858114 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 09:50:01.858127 (+ 13us) server_negotiation.cc:407] Connection header received
0812 09:50:01.858156 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:50:01.858159 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:50:01.858196 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:50:01.858284 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:50:01.858730 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:50:01.859975 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:50:01.861039 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:50:01.861399 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:50:01.861837 (+ 438us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:50:01.864379 (+ 2542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:50:01.864384 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:50:01.864388 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:50:01.864412 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:50:01.867150 (+ 2738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:50:01.867675 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:50:01.867677 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:50:01.867679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:50:01.867763 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:50:01.868109 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:50:01.868113 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:50:01.868116 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:50:01.868330 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:50:01.869182 (+ 852us) server_negotiation.cc:1035] Waiting for connection context
0812 09:50:01.869265 (+ 83us) server_negotiation.cc:299] Negotiation successful
0812 09:50:01.869373 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":138,"threads_started":1}
I20260812 09:50:01.869885 19272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:50:01.859886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.60:33401 (local address 127.0.0.1:46886)
0812 09:50:01.860213 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0812 09:50:01.860228 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 09:50:01.860299 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:50:01.860636 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:50:01.860641 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:50:01.860826 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:50:01.860957 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:50:01.860968 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:50:01.862218 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:50:01.862224 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:50:01.862911 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:50:01.862921 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:50:01.863461 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:50:01.863466 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:50:01.863571 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:50:01.864008 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:50:01.864026 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:50:01.865611 (+ 1585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:50:01.868212 (+ 2601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:50:01.868215 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:50:01.868217 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:50:01.868537 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:50:01.868851 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:50:01.868856 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:50:01.868858 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:50:01.868960 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:50:01.869396 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:50:01.869400 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:50:01.869508 (+ 108us) client_negotiation.cc:769] Sending connection context
0812 09:50:01.869601 (+ 93us) client_negotiation.cc:240] Negotiation successful
0812 09:50:01.869726 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252}
I20260812 09:50:01.870374 19400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:50:01.860137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46886 (local address 127.13.99.60:33401)
0812 09:50:01.860379 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0812 09:50:01.860395 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 09:50:01.860406 (+ 11us) server_negotiation.cc:407] Connection header received
0812 09:50:01.860436 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:50:01.860439 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:50:01.860497 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:50:01.861082 (+ 585us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:50:01.861221 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:50:01.862074 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:50:01.863035 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:50:01.863325 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:50:01.863801 (+ 476us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:50:01.865744 (+ 1943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:50:01.865749 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:50:01.865754 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:50:01.865780 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:50:01.868095 (+ 2315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:50:01.868650 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:50:01.868653 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:50:01.868655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:50:01.868738 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:50:01.869077 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:50:01.869080 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:50:01.869081 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:50:01.869277 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:50:01.870052 (+ 775us) server_negotiation.cc:1035] Waiting for connection context
0812 09:50:01.870123 (+ 71us) server_negotiation.cc:299] Negotiation successful
0812 09:50:01.870223 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"spinlock_wait_cycles":768,"thread_start_us":83,"threads_started":1}
I20260812 09:50:01.872008 19393 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c18c1248371649e3945385c9eebc937e: Bootstrap starting.
I20260812 09:50:01.872355 19146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 688b60ed30ec4f4488cceb1a2671ddcc: Bootstrap starting.
I20260812 09:50:01.873135 19266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1da86a8a31214e70aac5ada3147c94ed: Bootstrap starting.
I20260812 09:50:01.874589 19146 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 688b60ed30ec4f4488cceb1a2671ddcc: Neither blocks nor log segments found. Creating new log.
I20260812 09:50:01.874737 19393 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c18c1248371649e3945385c9eebc937e: Neither blocks nor log segments found. Creating new log.
I20260812 09:50:01.875378 19146 log.cc:824] T 00000000000000000000000000000000 P 688b60ed30ec4f4488cceb1a2671ddcc: Log is configured to *not* fsync() on all Append() calls
I20260812 09:50:01.875479 19266 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1da86a8a31214e70aac5ada3147c94ed: Neither blocks nor log segments found. Creating new log.
I20260812 09:50:01.875543 19393 log.cc:824] T 00000000000000000000000000000000 P c18c1248371649e3945385c9eebc937e: Log is configured to *not* fsync() on all Append() calls
I20260812 09:50:01.876412 19266 log.cc:824] T 00000000000000000000000000000000 P 1da86a8a31214e70aac5ada3147c94ed: Log is configured to *not* fsync() on all Append() calls
I20260812 09:50:01.877665 19146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 688b60ed30ec4f4488cceb1a2671ddcc: No bootstrap required, opened a new log
I20260812 09:50:01.877810 19393 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c18c1248371649e3945385c9eebc937e: No bootstrap required, opened a new log
I20260812 09:50:01.878408 19266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1da86a8a31214e70aac5ada3147c94ed: No bootstrap required, opened a new log
I20260812 09:50:01.881259 19393 raft_consensus.cc:348] T 00000000000000000000000000000000 P c18c1248371649e3945385c9eebc937e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "688b60ed30ec4f4488cceb1a2671ddcc" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 38981 } } peers { permanent_uuid: "1da86a8a31214e70aac5ada3147c94ed" member_type: VOTER last_known_addr { host: "127.13.99.61" port: 38005 } } peers { permanent_uuid: "c18c1248371649e3945385c9eebc937e" member_type: VOTER last_known_addr { host: "127.13.99.60" port: 33401 } }
I20260812 09:50:01.881525 19393 raft_consensus.cc:729] T 00000000000000000000000000000000 P c18c1248371649e3945385c9eebc937e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18c1248371649e3945385c9eebc937e, State: Initialized, Role: FOLLOWER
I20260812 09:50:01.881886 19393 consensus_queue.cc:260] T 00000000000000000000000000000000 P c18c1248371649e3945385c9eebc937e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "688b60ed30ec4f4488cceb1a2671ddcc" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 38981 } } peers { permanent_uuid: "1da86a8a31214e70aac5ada3147c94ed" member_type: VOTER last_known_addr { host: "127.13.99.61" port: 38005 } } peers { permanent_uuid: "c18c1248371649e3945385c9eebc937e" member_type: VOTER last_known_addr { host: "127.13.99.60" port: 33401 } }
I20260812 09:50:01.882156 19266 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1da86a8a31214e70aac5ada3147c94ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "688b60ed30ec4f4488cceb1a2671ddcc" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 38981 } } peers { permanent_uuid: "1da86a8a31214e70aac5ada3147c94ed" member_type: VOTER last_known_addr { host: "127.13.99.61" port: 38005 } } peers { permanent_uuid: "c18c1248371649e3945385c9eebc937e" member_type: VOTER last_known_addr { host: "127.13.99.60" port: 33401 } }
I20260812 09:50:01.882375 19266 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1da86a8a31214e70aac5ada3147c94ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1da86a8a31214e70aac5ada3147c94ed, State: Initialized, Role: FOLLOWER
I20260812 09:50:01.882251 19146 raft_consensus.cc:348] T 00000000000000000000000000000000 P 688b60ed30ec4f4488cceb1a2671ddcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "688b60ed30ec4f4488cceb1a2671ddcc" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 38981 } } peers { permanent_uuid: "1da86a8a31214e70aac5ada3147c94ed" member_type: VOTER last_known_addr { host: "127.13.99.61" port: 38005 } } peers { permanent_uuid: "c18c1248371649e3945385c9eebc937e" member_type: VOTER last_known_addr { host: "127.13.99.60" port: 33401 } }
I20260812 09:50:01.882597 19146 raft_consensus.cc:729] T 00000000000000000000000000000000 P 688b60ed30ec4f4488cceb1a2671ddcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 688b60ed30ec4f4488cceb1a2671ddcc, State: Initialized, Role: FOLLOWER
I20260812 09:50:01.882697 19266 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1da86a8a31214e70aac5ada3147c94ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "688b60ed30ec4f4488cceb1a2671ddcc" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 38981 } } peers { permanent_uuid: "1da86a8a31214e70aac5ada3147c94ed" member_type: VOTER last_known_addr { host: "127.13.99.61" port: 38005 } } peers { permanent_uuid: "c18c1248371649e3945385c9eebc937e" member_type: VOTER last_known_addr { host: "127.13.99.60" port: 33401 } }
I20260812 09:50:01.882969 19146 consensus_queue.cc:260] T 00000000000000000000000000000000 P 688b60ed30ec4f4488cceb1a2671ddcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "688b60ed30ec4f4488cceb1a2671ddcc" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 38981 } } peers { permanent_uuid: "1da86a8a31214e70aac5ada3147c94ed" member_type: VOTER last_known_addr { host: "127.13.99.61" port: 38005 } } peers { permanent_uuid: "c18c1248371649e3945385c9eebc937e" member_type: VOTER last_known_addr { host: "127.13.99.60" port: 33401 } }
I20260812 09:50:01.882990 19408 sys_catalog.cc:455] T 00000000000000000000000000000000 P c18c1248371649e3945385c9eebc937e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "688b60ed30ec4f4488cceb1a2671ddcc" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 38981 } } peers { permanent_uuid: "1da86a8a31214e70aac5ada3147c94ed" member_type: VOTER last_known_addr { host: "127.13.99.61" port: 38005 } } peers { permanent_uuid: "c18c1248371649e3945385c9eebc937e" member_type: VOTER last_known_addr { host: "127.13.99.60" port: 33401 } } }
I20260812 09:50:01.883119 19408 sys_catalog.cc:458] T 00000000000000000000000000000000 P c18c1248371649e3945385c9eebc937e [sys.catalog]: This master's current role is: FOLLOWER
I20260812 09:50:01.883301 19393 sys_catalog.cc:564] T 00000000000000000000000000000000 P c18c1248371649e3945385c9eebc937e [sys.catalog]: configured and running, proceeding with master startup.
I20260812 09:50:01.883373 19409 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1da86a8a31214e70aac5ada3147c94ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "688b60ed30ec4f4488cceb1a2671ddcc" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 38981 } } peers { permanent_uuid: "1da86a8a31214e70aac5ada3147c94ed" member_type: VOTER last_known_addr { host: "127.13.99.61" port: 38005 } } peers { permanent_uuid: "c18c1248371649e3945385c9eebc937e" member_type: VOTER last_known_addr { host: "127.13.99.60" port: 33401 } } }
I20260812 09:50:01.883524 19409 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1da86a8a31214e70aac5ada3147c94ed [sys.catalog]: This master's current role is: FOLLOWER
I20260812 09:50:01.883790 19410 sys_catalog.cc:455] T 00000000000000000000000000000000 P 688b60ed30ec4f4488cceb1a2671ddcc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "688b60ed30ec4f4488cceb1a2671ddcc" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 38981 } } peers { permanent_uuid: "1da86a8a31214e70aac5ada3147c94ed" member_type: VOTER last_known_addr { host: "127.13.99.61" port: 38005 } } peers { permanent_uuid: "c18c1248371649e3945385c9eebc937e" member_type: VOTER last_known_addr { host: "127.13.99.60" port: 33401 } } }
I20260812 09:50:01.883922 19410 sys_catalog.cc:458] T 00000000000000000000000000000000 P 688b60ed30ec4f4488cceb1a2671ddcc [sys.catalog]: This master's current role is: FOLLOWER
I20260812 09:50:01.883961 19266 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1da86a8a31214e70aac5ada3147c94ed [sys.catalog]: configured and running, proceeding with master startup.
I20260812 09:50:01.884593 19146 sys_catalog.cc:564] T 00000000000000000000000000000000 P 688b60ed30ec4f4488cceb1a2671ddcc [sys.catalog]: configured and running, proceeding with master startup.
I20260812 09:50:01.886565 13708 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 09:50:01.888397 19105 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "688b60ed30ec4f4488cceb1a2671ddcc"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35534
I20260812 09:50:01.888764 19105 raft_consensus.cc:482] T 00000000000000000000000000000000 P 688b60ed30ec4f4488cceb1a2671ddcc [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 09:50:01.888881 19105 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 688b60ed30ec4f4488cceb1a2671ddcc [term 0 FOLLOWER]: Advancing to term 1
W20260812 09:50:01.888824 19441 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c18c1248371649e3945385c9eebc937e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 09:50:01.889036 19441 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 09:50:01.887117 19279 master_runner.cc:186] Error getting master registration for 127.13.99.62:38981: OK, instance_id { permanent_uuid: "688b60ed30ec4f4488cceb1a2671ddcc" instance_seqno: 1786528201390508 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
W20260812 09:50:01.889864 19455 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 688b60ed30ec4f4488cceb1a2671ddcc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 09:50:01.889940 19455 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 09:50:01.890039 19279 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 09:50:01.890280 19105 raft_consensus.cc:504] T 00000000000000000000000000000000 P 688b60ed30ec4f4488cceb1a2671ddcc [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "688b60ed30ec4f4488cceb1a2671ddcc" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 38981 } } peers { permanent_uuid: "1da86a8a31214e70aac5ada3147c94ed" member_type: VOTER last_known_addr { host: "127.13.99.61" port: 38005 } } peers { permanent_uuid: "c18c1248371649e3945385c9eebc937e" member_type: VOTER last_known_addr { host: "127.13.99.60" port: 33401 } }
I20260812 09:50:01.891197 19105 leader_election.cc:288] T 00000000000000000000000000000000 P 688b60ed30ec4f4488cceb1a2671ddcc [CANDIDATE]: Term 1 election: Requested vote from peers 1da86a8a31214e70aac5ada3147c94ed (127.13.99.61:38005), c18c1248371649e3945385c9eebc937e (127.13.99.60:33401)
W20260812 09:50:01.887162 19074 tablet.cc:2369] T 00000000000000000000000000000000 P 688b60ed30ec4f4488cceb1a2671ddcc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 09:50:01.892627 19351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "688b60ed30ec4f4488cceb1a2671ddcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c18c1248371649e3945385c9eebc937e"
I20260812 09:50:01.892621 19224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "688b60ed30ec4f4488cceb1a2671ddcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "1da86a8a31214e70aac5ada3147c94ed"
I20260812 09:50:01.892843 19351 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c18c1248371649e3945385c9eebc937e [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:50:01.892850 19224 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1da86a8a31214e70aac5ada3147c94ed [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:50:01.893997 19224 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1da86a8a31214e70aac5ada3147c94ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 688b60ed30ec4f4488cceb1a2671ddcc in term 1.
I20260812 09:50:01.894189 19351 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c18c1248371649e3945385c9eebc937e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 688b60ed30ec4f4488cceb1a2671ddcc in term 1.
I20260812 09:50:01.894547 19063 leader_election.cc:302] T 00000000000000000000000000000000 P 688b60ed30ec4f4488cceb1a2671ddcc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1da86a8a31214e70aac5ada3147c94ed, 688b60ed30ec4f4488cceb1a2671ddcc; no voters:
I20260812 09:50:01.894846 19410 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 688b60ed30ec4f4488cceb1a2671ddcc [term 1 FOLLOWER]: Leader election won for term 1
I20260812 09:50:01.895056 19410 raft_consensus.cc:686] T 00000000000000000000000000000000 P 688b60ed30ec4f4488cceb1a2671ddcc [term 1 LEADER]: Becoming Leader. State: Replica: 688b60ed30ec4f4488cceb1a2671ddcc, State: Running, Role: LEADER
I20260812 09:50:01.895357 19410 consensus_queue.cc:237] T 00000000000000000000000000000000 P 688b60ed30ec4f4488cceb1a2671ddcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "688b60ed30ec4f4488cceb1a2671ddcc" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 38981 } } peers { permanent_uuid: "1da86a8a31214e70aac5ada3147c94ed" member_type: VOTER last_known_addr { host: "127.13.99.61" port: 38005 } } peers { permanent_uuid: "c18c1248371649e3945385c9eebc937e" member_type: VOTER last_known_addr { host: "127.13.99.60" port: 33401 } }
I20260812 09:50:01.897038 19410 sys_catalog.cc:455] T 00000000000000000000000000000000 P 688b60ed30ec4f4488cceb1a2671ddcc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 688b60ed30ec4f4488cceb1a2671ddcc. Latest consensus state: current_term: 1 leader_uuid: "688b60ed30ec4f4488cceb1a2671ddcc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "688b60ed30ec4f4488cceb1a2671ddcc" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 38981 } } peers { permanent_uuid: "1da86a8a31214e70aac5ada3147c94ed" member_type: VOTER last_known_addr { host: "127.13.99.61" port: 38005 } } peers { permanent_uuid: "c18c1248371649e3945385c9eebc937e" member_type: VOTER last_known_addr { host: "127.13.99.60" port: 33401 } } }
I20260812 09:50:01.897166 19410 sys_catalog.cc:458] T 00000000000000000000000000000000 P 688b60ed30ec4f4488cceb1a2671ddcc [sys.catalog]: This master's current role is: LEADER
I20260812 09:50:01.895193 19043 master_runner.cc:186] Error getting master registration for 127.13.99.61:38005: OK, instance_id { permanent_uuid: "1da86a8a31214e70aac5ada3147c94ed" instance_seqno: 1786528201601623 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 09:50:01.898542 19043 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260812 09:50:01.902029 19478 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1da86a8a31214e70aac5ada3147c94ed: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 09:50:01.902231 19478 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 09:50:01.911883 19477 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 09:50:01.912983 19477 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 09:50:01.919534 19149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:50:01.899893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35546 (local address 127.13.99.62:38981)
0812 09:50:01.900012 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0812 09:50:01.900018 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 09:50:01.908830 (+ 8812us) server_negotiation.cc:407] Connection header received
0812 09:50:01.909054 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:50:01.909058 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:50:01.909109 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:50:01.909163 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:50:01.910598 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:50:01.911423 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:50:01.913244 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:50:01.913654 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:50:01.913739 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:50:01.915767 (+ 2028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:50:01.915771 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:50:01.915776 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:50:01.915805 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:50:01.917620 (+ 1815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:50:01.918140 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:50:01.918144 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:50:01.918146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:50:01.918235 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:50:01.918550 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:50:01.918554 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:50:01.918556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:50:01.918768 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:50:01.918869 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0812 09:50:01.919245 (+ 376us) server_negotiation.cc:299] Negotiation successful
0812 09:50:01.919367 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260812 09:50:01.927563 19400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:50:01.902009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46898 (local address 127.13.99.60:33401)
0812 09:50:01.902129 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0812 09:50:01.902133 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:50:01.912260 (+ 10127us) server_negotiation.cc:407] Connection header received
0812 09:50:01.912398 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:50:01.912402 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:50:01.912449 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:50:01.912563 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:50:01.919920 (+ 7357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:50:01.920691 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:50:01.921807 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:50:01.922198 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:50:01.922273 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:50:01.923927 (+ 1654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:50:01.923931 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:50:01.923934 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:50:01.923963 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:50:01.925733 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:50:01.926213 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:50:01.926217 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:50:01.926218 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:50:01.926300 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:50:01.926599 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:50:01.926603 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:50:01.926604 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:50:01.926810 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:50:01.926919 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0812 09:50:01.927268 (+ 349us) server_negotiation.cc:299] Negotiation successful
0812 09:50:01.927386 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260812 09:50:01.929903 19278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:50:01.900708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59568 (local address 127.13.99.61:38005)
0812 09:50:01.900818 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0812 09:50:01.900822 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:50:01.910439 (+ 9617us) server_negotiation.cc:407] Connection header received
0812 09:50:01.910550 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:50:01.910553 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:50:01.910613 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:50:01.910701 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:50:01.915305 (+ 4604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:50:01.916087 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:50:01.917263 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:50:01.917604 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:50:01.917685 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:50:01.919343 (+ 1658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:50:01.919347 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:50:01.919350 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:50:01.919376 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:50:01.921104 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:50:01.928572 (+ 7468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:50:01.928576 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:50:01.928578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:50:01.928670 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:50:01.928986 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:50:01.928989 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:50:01.928991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:50:01.929196 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:50:01.929314 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0812 09:50:01.929640 (+ 326us) server_negotiation.cc:299] Negotiation successful
0812 09:50:01.929753 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260812 09:50:01.932950 19224 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1da86a8a31214e70aac5ada3147c94ed [term 1 FOLLOWER]: Refusing update from remote peer 688b60ed30ec4f4488cceb1a2671ddcc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 09:50:01.933275 19351 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c18c1248371649e3945385c9eebc937e [term 1 FOLLOWER]: Refusing update from remote peer 688b60ed30ec4f4488cceb1a2671ddcc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 09:50:01.933810 19410 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 688b60ed30ec4f4488cceb1a2671ddcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "1da86a8a31214e70aac5ada3147c94ed" member_type: VOTER last_known_addr { host: "127.13.99.61" port: 38005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 09:50:01.934039 19469 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 688b60ed30ec4f4488cceb1a2671ddcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c18c1248371649e3945385c9eebc937e" member_type: VOTER last_known_addr { host: "127.13.99.60" port: 33401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 09:50:01.937844 19409 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1da86a8a31214e70aac5ada3147c94ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 688b60ed30ec4f4488cceb1a2671ddcc. Latest consensus state: current_term: 1 leader_uuid: "688b60ed30ec4f4488cceb1a2671ddcc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "688b60ed30ec4f4488cceb1a2671ddcc" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 38981 } } peers { permanent_uuid: "1da86a8a31214e70aac5ada3147c94ed" member_type: VOTER last_known_addr { host: "127.13.99.61" port: 38005 } } peers { permanent_uuid: "c18c1248371649e3945385c9eebc937e" member_type: VOTER last_known_addr { host: "127.13.99.60" port: 33401 } } }
I20260812 09:50:01.937973 19409 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1da86a8a31214e70aac5ada3147c94ed [sys.catalog]: This master's current role is: FOLLOWER
I20260812 09:50:01.943375 19408 sys_catalog.cc:455] T 00000000000000000000000000000000 P c18c1248371649e3945385c9eebc937e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 688b60ed30ec4f4488cceb1a2671ddcc. Latest consensus state: current_term: 1 leader_uuid: "688b60ed30ec4f4488cceb1a2671ddcc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "688b60ed30ec4f4488cceb1a2671ddcc" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 38981 } } peers { permanent_uuid: "1da86a8a31214e70aac5ada3147c94ed" member_type: VOTER last_known_addr { host: "127.13.99.61" port: 38005 } } peers { permanent_uuid: "c18c1248371649e3945385c9eebc937e" member_type: VOTER last_known_addr { host: "127.13.99.60" port: 33401 } } }
I20260812 09:50:01.943495 19408 sys_catalog.cc:458] T 00000000000000000000000000000000 P c18c1248371649e3945385c9eebc937e [sys.catalog]: This master's current role is: FOLLOWER
I20260812 09:50:01.953995 19469 sys_catalog.cc:455] T 00000000000000000000000000000000 P 688b60ed30ec4f4488cceb1a2671ddcc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "688b60ed30ec4f4488cceb1a2671ddcc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "688b60ed30ec4f4488cceb1a2671ddcc" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 38981 } } peers { permanent_uuid: "1da86a8a31214e70aac5ada3147c94ed" member_type: VOTER last_known_addr { host: "127.13.99.61" port: 38005 } } peers { permanent_uuid: "c18c1248371649e3945385c9eebc937e" member_type: VOTER last_known_addr { host: "127.13.99.60" port: 33401 } } }
I20260812 09:50:01.954144 19469 sys_catalog.cc:458] T 00000000000000000000000000000000 P 688b60ed30ec4f4488cceb1a2671ddcc [sys.catalog]: This master's current role is: LEADER
I20260812 09:50:01.954499 19485 mvcc.cc:205] Tried to move back new op lower bound from 7317619515109789696 to 7317619514966319104. Current Snapshot: MvccSnapshot[applied={T|T < 7317619515109789696}]
I20260812 09:50:01.955571 19408 sys_catalog.cc:455] T 00000000000000000000000000000000 P c18c1248371649e3945385c9eebc937e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "688b60ed30ec4f4488cceb1a2671ddcc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "688b60ed30ec4f4488cceb1a2671ddcc" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 38981 } } peers { permanent_uuid: "1da86a8a31214e70aac5ada3147c94ed" member_type: VOTER last_known_addr { host: "127.13.99.61" port: 38005 } } peers { permanent_uuid: "c18c1248371649e3945385c9eebc937e" member_type: VOTER last_known_addr { host: "127.13.99.60" port: 33401 } } }
I20260812 09:50:01.955686 19408 sys_catalog.cc:458] T 00000000000000000000000000000000 P c18c1248371649e3945385c9eebc937e [sys.catalog]: This master's current role is: FOLLOWER
I20260812 09:50:01.958151 19410 sys_catalog.cc:455] T 00000000000000000000000000000000 P 688b60ed30ec4f4488cceb1a2671ddcc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "688b60ed30ec4f4488cceb1a2671ddcc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "688b60ed30ec4f4488cceb1a2671ddcc" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 38981 } } peers { permanent_uuid: "1da86a8a31214e70aac5ada3147c94ed" member_type: VOTER last_known_addr { host: "127.13.99.61" port: 38005 } } peers { permanent_uuid: "c18c1248371649e3945385c9eebc937e" member_type: VOTER last_known_addr { host: "127.13.99.60" port: 33401 } } }
I20260812 09:50:01.958267 19410 sys_catalog.cc:458] T 00000000000000000000000000000000 P 688b60ed30ec4f4488cceb1a2671ddcc [sys.catalog]: This master's current role is: LEADER
I20260812 09:50:01.959306 19409 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1da86a8a31214e70aac5ada3147c94ed [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "688b60ed30ec4f4488cceb1a2671ddcc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "688b60ed30ec4f4488cceb1a2671ddcc" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 38981 } } peers { permanent_uuid: "1da86a8a31214e70aac5ada3147c94ed" member_type: VOTER last_known_addr { host: "127.13.99.61" port: 38005 } } peers { permanent_uuid: "c18c1248371649e3945385c9eebc937e" member_type: VOTER last_known_addr { host: "127.13.99.60" port: 33401 } } }
I20260812 09:50:01.959420 19409 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1da86a8a31214e70aac5ada3147c94ed [sys.catalog]: This master's current role is: FOLLOWER
I20260812 09:50:01.957520 19489 mvcc.cc:205] Tried to move back new op lower bound from 7317619515109789696 to 7317619514966319104. Current Snapshot: MvccSnapshot[applied={T|T < 7317619515109789696}]
I20260812 09:50:01.962302 19477 catalog_manager.cc:1347] Generated new cluster ID: f8a9b54f3b46411fbe6bf37599dee866
I20260812 09:50:01.962412 19477 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 09:50:01.963171 19486 mvcc.cc:205] Tried to move back new op lower bound from 7317619515109789696 to 7317619514966319104. Current Snapshot: MvccSnapshot[applied={T|T < 7317619515109789696}]
I20260812 09:50:01.995839 19477 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 09:50:01.997184 19477 catalog_manager.cc:1504] Loading token signing keys...
I20260812 09:50:02.014760 19477 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 688b60ed30ec4f4488cceb1a2671ddcc: Generated new TSK 0
I20260812 09:50:02.015769 19477 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 09:50:02.047708 19149 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 09:50:02.045292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35558 (local address 127.13.99.62:38981)
0812 09:50:02.045450 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0812 09:50:02.045455 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:50:02.046535 (+ 1080us) server_negotiation.cc:407] Connection header received
0812 09:50:02.046754 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:50:02.046758 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:50:02.046813 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:50:02.046871 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:50:02.047572 (+ 701us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35558: BlockingRecv error: recv got EOF from 127.0.0.1:35558 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260812 09:50:02.049629 19278 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 09:50:02.045770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59580 (local address 127.13.99.61:38005)
0812 09:50:02.045909 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0812 09:50:02.045914 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:50:02.048235 (+ 2321us) server_negotiation.cc:407] Connection header received
0812 09:50:02.048428 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:50:02.048432 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:50:02.048533 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:50:02.048592 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:50:02.049508 (+ 916us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59580: BlockingRecv error: recv got EOF from 127.0.0.1:59580 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260812 09:50:02.050745 19400 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 09:50:02.046092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46904 (local address 127.13.99.60:33401)
0812 09:50:02.046237 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0812 09:50:02.046241 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:50:02.048892 (+ 2651us) server_negotiation.cc:407] Connection header received
0812 09:50:02.050007 (+ 1115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:50:02.050010 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:50:02.050062 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:50:02.050115 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:50:02.050630 (+ 515us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46904: BlockingRecv error: recv got EOF from 127.0.0.1:46904 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260812 09:50:02.055451 19278 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 09:50:02.053977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59594 (local address 127.13.99.61:38005)
0812 09:50:02.054080 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0812 09:50:02.054084 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:50:02.054517 (+ 433us) server_negotiation.cc:407] Connection header received
0812 09:50:02.054731 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:50:02.054735 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:50:02.054809 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:50:02.054860 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:50:02.055347 (+ 487us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59594: BlockingRecv error: recv got EOF from 127.0.0.1:59594 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260812 09:50:02.060657 19278 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 09:50:02.059055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59598 (local address 127.13.99.61:38005)
0812 09:50:02.059177 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0812 09:50:02.059181 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:50:02.059668 (+ 487us) server_negotiation.cc:407] Connection header received
0812 09:50:02.059865 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:50:02.059868 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:50:02.059964 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:50:02.060022 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:50:02.060567 (+ 545us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59598: BlockingRecv error: recv got EOF from 127.0.0.1:59598 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260812 09:50:02.066493 19278 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 09:50:02.064955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59604 (local address 127.13.99.61:38005)
0812 09:50:02.065063 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0812 09:50:02.065067 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:50:02.065551 (+ 484us) server_negotiation.cc:407] Connection header received
0812 09:50:02.065749 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:50:02.065752 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:50:02.065806 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:50:02.065856 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:50:02.066391 (+ 535us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59604: BlockingRecv error: recv got EOF from 127.0.0.1:59604 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260812 09:50:02.074743 19278 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 09:50:02.072965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59616 (local address 127.13.99.61:38005)
0812 09:50:02.073103 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0812 09:50:02.073107 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:50:02.073747 (+ 640us) server_negotiation.cc:407] Connection header received
0812 09:50:02.073978 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:50:02.073982 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:50:02.074042 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:50:02.074098 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:50:02.074640 (+ 542us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59616: BlockingRecv error: recv got EOF from 127.0.0.1:59616 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260812 09:50:02.086957 19278 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 09:50:02.085303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59628 (local address 127.13.99.61:38005)
0812 09:50:02.085433 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0812 09:50:02.085437 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:50:02.085964 (+ 527us) server_negotiation.cc:407] Connection header received
0812 09:50:02.086200 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:50:02.086204 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:50:02.086250 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:50:02.086298 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:50:02.086844 (+ 546us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59628: BlockingRecv error: recv got EOF from 127.0.0.1:59628 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260812 09:50:02.106950 19278 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 09:50:02.105364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59634 (local address 127.13.99.61:38005)
0812 09:50:02.105479 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0812 09:50:02.105483 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:50:02.106013 (+ 530us) server_negotiation.cc:407] Connection header received
0812 09:50:02.106207 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:50:02.106211 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:50:02.106262 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:50:02.106310 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:50:02.106828 (+ 518us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59634: BlockingRecv error: recv got EOF from 127.0.0.1:59634 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260812 09:50:02.142827 19278 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 09:50:02.141588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59638 (local address 127.13.99.61:38005)
0812 09:50:02.141722 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0812 09:50:02.141726 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:50:02.141912 (+ 186us) server_negotiation.cc:407] Connection header received
0812 09:50:02.142122 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:50:02.142125 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:50:02.142185 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:50:02.142284 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:50:02.142691 (+ 407us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59638: BlockingRecv error: recv got EOF from 127.0.0.1:59638 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260812 09:50:02.210547 19278 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 09:50:02.209415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59652 (local address 127.13.99.61:38005)
0812 09:50:02.209594 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0812 09:50:02.209598 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:50:02.209714 (+ 116us) server_negotiation.cc:407] Connection header received
0812 09:50:02.209844 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:50:02.209847 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:50:02.209896 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:50:02.209981 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:50:02.210446 (+ 465us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59652: BlockingRecv error: recv got EOF from 127.0.0.1:59652 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260812 09:50:02.342067 19278 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 09:50:02.340885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59666 (local address 127.13.99.61:38005)
0812 09:50:02.341050 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0812 09:50:02.341056 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 09:50:02.341215 (+ 159us) server_negotiation.cc:407] Connection header received
0812 09:50:02.341343 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:50:02.341346 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:50:02.341398 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:50:02.341475 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:50:02.341929 (+ 454us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59666: BlockingRecv error: recv got EOF from 127.0.0.1:59666 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260812 09:50:02.601408 19278 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 09:50:02.600296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59682 (local address 127.13.99.61:38005)
0812 09:50:02.600432 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0812 09:50:02.600436 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:50:02.600497 (+ 61us) server_negotiation.cc:407] Connection header received
0812 09:50:02.600663 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:50:02.600666 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:50:02.600717 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:50:02.600802 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:50:02.601257 (+ 455us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59682: BlockingRecv error: recv got EOF from 127.0.0.1:59682 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260812 09:50:02.890465 19441 catalog_manager.cc:1259] Loaded cluster ID: f8a9b54f3b46411fbe6bf37599dee866
I20260812 09:50:02.890555 19441 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c18c1248371649e3945385c9eebc937e: loading cluster ID for follower catalog manager: success
I20260812 09:50:02.892268 19441 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c18c1248371649e3945385c9eebc937e: acquiring CA information for follower catalog manager: success
I20260812 09:50:02.893648 19441 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c18c1248371649e3945385c9eebc937e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 09:50:02.903304 19478 catalog_manager.cc:1259] Loaded cluster ID: f8a9b54f3b46411fbe6bf37599dee866
I20260812 09:50:02.903402 19478 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1da86a8a31214e70aac5ada3147c94ed: loading cluster ID for follower catalog manager: success
I20260812 09:50:02.905022 19478 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1da86a8a31214e70aac5ada3147c94ed: acquiring CA information for follower catalog manager: success
I20260812 09:50:02.906178 19478 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1da86a8a31214e70aac5ada3147c94ed: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 09:50:03.119843 19604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:50:03.115773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59694 (local address 127.13.99.61:38005)
0812 09:50:03.116049 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0812 09:50:03.116056 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 09:50:03.116080 (+ 24us) server_negotiation.cc:407] Connection header received
0812 09:50:03.116290 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:50:03.116293 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:50:03.116427 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:50:03.116582 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 09:50:03.116866 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:50:03.117390 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:50:03.118283 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:50:03.118517 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:50:03.118634 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:50:03.118961 (+ 327us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 09:50:03.119406 (+ 445us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 09:50:03.119550 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0812 09:50:03.119621 (+ 71us) server_negotiation.cc:299] Negotiation successful
0812 09:50:03.119681 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":83,"threads_started":1}
I20260812 09:50:03.126211 19611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:50:03.122845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46916 (local address 127.13.99.60:33401)
0812 09:50:03.123058 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0812 09:50:03.123062 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:50:03.123075 (+ 13us) server_negotiation.cc:407] Connection header received
0812 09:50:03.123189 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:50:03.123191 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:50:03.123294 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:50:03.123388 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 09:50:03.123583 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:50:03.124086 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:50:03.124909 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:50:03.125143 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:50:03.125224 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:50:03.125467 (+ 243us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 09:50:03.125828 (+ 361us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 09:50:03.125914 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0812 09:50:03.125971 (+ 57us) server_negotiation.cc:299] Negotiation successful
0812 09:50:03.126015 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":64,"threads_started":1}
I20260812 09:50:03.133183 19611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:50:03.129809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46924 (local address 127.13.99.60:33401)
0812 09:50:03.129931 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0812 09:50:03.129937 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 09:50:03.130059 (+ 122us) server_negotiation.cc:407] Connection header received
0812 09:50:03.130274 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:50:03.130277 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:50:03.130326 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:50:03.130404 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 09:50:03.130643 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:50:03.131169 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:50:03.132051 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:50:03.132337 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:50:03.132432 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:50:03.132674 (+ 242us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 09:50:03.132757 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 09:50:03.132825 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0812 09:50:03.132988 (+ 163us) server_negotiation.cc:299] Negotiation successful
0812 09:50:03.133035 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260812 09:50:03.133674 19604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:50:03.129456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59702 (local address 127.13.99.61:38005)
0812 09:50:03.129568 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0812 09:50:03.129572 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:50:03.129797 (+ 225us) server_negotiation.cc:407] Connection header received
0812 09:50:03.129980 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:50:03.129983 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:50:03.130033 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:50:03.130078 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 09:50:03.130460 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:50:03.131194 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:50:03.132369 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:50:03.132765 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:50:03.132860 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:50:03.133188 (+ 328us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 09:50:03.133257 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 09:50:03.133302 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0812 09:50:03.133502 (+ 200us) server_negotiation.cc:299] Negotiation successful
0812 09:50:03.133559 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260812 09:50:03.133802 19619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:50:03.129171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35562 (local address 127.13.99.62:38981)
0812 09:50:03.129403 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0812 09:50:03.129408 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:50:03.129424 (+ 16us) server_negotiation.cc:407] Connection header received
0812 09:50:03.129519 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:50:03.129521 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:50:03.129681 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:50:03.129809 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 09:50:03.130187 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:50:03.131018 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:50:03.132276 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:50:03.132655 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:50:03.132764 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:50:03.133100 (+ 336us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 09:50:03.133394 (+ 294us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 09:50:03.133482 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0812 09:50:03.133594 (+ 112us) server_negotiation.cc:299] Negotiation successful
0812 09:50:03.133648 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":82,"threads_started":1}
I20260812 09:50:03.136818 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 19043
I20260812 09:50:03.154554 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 19154
I20260812 09:50:03.165730 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 19279
2026-08-12T09:50:03Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3350 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786527905902985-13708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 09:50:03 dist-test-slave-jq4t krb5kdc[19629](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 09:50:03 dist-test-slave-jq4t krb5kdc[19629](info): set up 2 sockets
Aug 12 09:50:03 dist-test-slave-jq4t krb5kdc[19629](info): commencing operation
krb5kdc: starting...
W20260812 09:50:04.442973 13708 mini_kdc.cc:121] Time spent starting KDC: real 1.249s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786527905902985-13708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786527905902985-13708-0/krb5kdc/test-user.keytab.
Aug 12 09:50:04 dist-test-slave-jq4t krb5kdc[19629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528204, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T09:50:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T09:50:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.99.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.99.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.99.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab.
Entry for principal kudu/127.13.99.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab.
I20260812 09:50:04.573776 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786527905902985-13708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786527905902985-13708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786527905902985-13708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.62:45545
--webserver_interface=127.13.99.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab
--principal=kudu/127.13.99.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:45301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.99.62:45545,127.13.99.61:39371,127.13.99.60:37813
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 09:50:04.689584 19649 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 09:50:04.689841 19649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:50:04.689864 19649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:50:04.689900 19649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:50:04.693878 19649 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 09:50:04.693931 19649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:50:04.693950 19649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:50:04.694012 19649 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 09:50:04.694049 19649 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 09:50:04.699296 19649 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:45301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786527905902985-13708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786527905902985-13708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.99.62:45545,127.13.99.61:39371,127.13.99.60:37813
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.62.keytab
--principal=kudu/127.13.99.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.62:45545
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.99.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19649
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786527905902985-13708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:50:04.700368 19649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:50:04.701421 19649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:50:04.709896 19665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:50:04.709914 19660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:50:04.710109 19649 server_base.cc:1034] running on GCE node
W20260812 09:50:04.709909 19659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:50:04.710795 19649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:50:04.711843 19649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:50:04.713032 19649 hybrid_clock.cc:648] HybridClock initialized: now 1786528204713003 us; error 39 us; skew 500 ppm
Aug 12 09:50:04 dist-test-slave-jq4t krb5kdc[19629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528204, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:50:04.716233 19649 init.cc:377] Logged in from keytab as kudu/127.13.99.62@KRBTEST.COM (short username kudu)
I20260812 09:50:04.717206 19649 webserver.cc:460] Webserver started at http://127.13.99.62:42569/ using document root <none> and password file <none>
I20260812 09:50:04.717795 19649 fs_manager.cc:359] Metadata directory not provided
I20260812 09:50:04.717892 19649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:50:04.718159 19649 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:50:04.720884 19649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786527905902985-13708-0/minicluster-data/master-0/data/instance:
uuid: "7bb031a243384595910e3baae543a5ac"
format_stamp: "Formatted at 2026-08-12 09:50:04 on dist-test-slave-jq4t"
server_key: "225b6d4dfccd29436a89810c99ab870d"
server_key_iv: "8bc47a8953c68d6a0895a4faeaf00ad3"
server_key_version: "encryptionkey@0"
I20260812 09:50:04.721601 19649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786527905902985-13708-0/minicluster-data/master-0/wal/instance:
uuid: "7bb031a243384595910e3baae543a5ac"
format_stamp: "Formatted at 2026-08-12 09:50:04 on dist-test-slave-jq4t"
server_key: "225b6d4dfccd29436a89810c99ab870d"
server_key_iv: "8bc47a8953c68d6a0895a4faeaf00ad3"
server_key_version: "encryptionkey@0"
I20260812 09:50:04.725538 19649 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260812 09:50:04.728055 19681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:50:04.729477 19649 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 09:50:04.729620 19649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786527905902985-13708-0/minicluster-data/master-0/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786527905902985-13708-0/minicluster-data/master-0/wal
uuid: "7bb031a243384595910e3baae543a5ac"
format_stamp: "Formatted at 2026-08-12 09:50:04 on dist-test-slave-jq4t"
server_key: "225b6d4dfccd29436a89810c99ab870d"
server_key_iv: "8bc47a8953c68d6a0895a4faeaf00ad3"
server_key_version: "encryptionkey@0"
I20260812 09:50:04.729727 19649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786527905902985-13708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786527905902985-13708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786527905902985-13708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:50:04.750253 19649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:50:04.753371 19649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:50:04.753537 19649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:50:04.760699 19649 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.62:45545
I20260812 09:50:04.761016 19757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.62:45545 every 8 connection(s)
I20260812 09:50:04.761731 19649 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786527905902985-13708-0/minicluster-data/master-0/data/info.pb
I20260812 09:50:04.764631 19758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:50:04.768365 19758 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.99.62" port: 45545 } has no permanent_uuid. Determining permanent_uuid...
I20260812 09:50:04.770963 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 19649
I20260812 09:50:04.771099 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786527905902985-13708-0/minicluster-data/master-0/wal/instance
I20260812 09:50:04.771423 13708 external_mini_cluster.cc:1428] Setting key 08714767d6e7036940a3ab26b381ad27
Aug 12 09:50:04 dist-test-slave-jq4t krb5kdc[19629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528204, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.62@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.99.61@KRBTEST.COM; defaulting to no policy
I20260812 09:50:04.784497 19760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:50:04.770608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:45545 (local address 127.0.0.1:43244)
0812 09:50:04.771148 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0812 09:50:04.771200 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0812 09:50:04.772091 (+ 891us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:50:04.772958 (+ 867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:50:04.772962 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:50:04.773303 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:50:04.773556 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:50:04.773574 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:50:04.774755 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:50:04.774758 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:50:04.775488 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:50:04.775497 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:50:04.776406 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:50:04.776409 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:50:04.776574 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:50:04.777167 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:50:04.777183 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:50:04.778680 (+ 1497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:50:04.781113 (+ 2433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:50:04.781130 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:50:04.781138 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:50:04.781371 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:50:04.781863 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:50:04.781866 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:50:04.781867 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:50:04.782007 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:50:04.782679 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:50:04.782686 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:50:04.782836 (+ 150us) client_negotiation.cc:769] Sending connection context
0812 09:50:04.783009 (+ 173us) client_negotiation.cc:240] Negotiation successful
0812 09:50:04.783440 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"mutex_wait_us":150,"thread_start_us":177,"threads_started":1}
I20260812 09:50:04.784502 19761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:50:04.770608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43244 (local address 127.13.99.62:45545)
0812 09:50:04.771177 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0812 09:50:04.771197 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0812 09:50:04.771410 (+ 213us) server_negotiation.cc:407] Connection header received
0812 09:50:04.772425 (+ 1015us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:50:04.772435 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:50:04.772736 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:50:04.772945 (+ 209us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:50:04.773721 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:50:04.774624 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:50:04.775628 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:50:04.776244 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:50:04.776434 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:50:04.778851 (+ 2417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:50:04.778861 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:50:04.778879 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:50:04.778912 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:50:04.780993 (+ 2081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:50:04.781505 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:50:04.781513 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:50:04.781515 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:50:04.781698 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:50:04.782134 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:50:04.782141 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:50:04.782146 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:50:04.782549 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:50:04.782710 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0812 09:50:04.783089 (+ 379us) server_negotiation.cc:299] Negotiation successful
0812 09:50:04.783389 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":442,"thread_start_us":153,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.99.61@KRBTEST.COM" created.
I20260812 09:50:04.787086 19758 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.99.61" port: 39371 } has no permanent_uuid. Determining permanent_uuid...
I20260812 09:50:04.787909 19760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:50:04.787612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.61:39371 (local address 127.0.0.1:47126)
0812 09:50:04.787760 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0812 09:50:04.787838 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.61:39371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260812 09:50:04.788209 19683 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.99.61:39371: connect: Connection refused (error 111)
I20260812 09:50:04.789228 19760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:50:04.788923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.61:39371 (local address 127.0.0.1:47132)
0812 09:50:04.789078 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0812 09:50:04.789142 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.61:39371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260812 09:50:04.789458 19758 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.99.61:39371: Network error: Client connection negotiation failed: client connection to 127.13.99.61:39371: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.99.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.61.keytab.
Entry for principal kudu/127.13.99.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.61.keytab.
I20260812 09:50:04.803681 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786527905902985-13708-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786527905902985-13708-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786527905902985-13708-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786527905902985-13708-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.61:39371
--webserver_interface=127.13.99.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.61.keytab
--principal=kudu/127.13.99.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:45301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.99.62:45545,127.13.99.61:39371,127.13.99.60:37813
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 09:50:04.838737 19758 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.99.61" port: 39371 } attempt: 1
I20260812 09:50:04.839671 19760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:50:04.839318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.61:39371 (local address 127.0.0.1:47140)
0812 09:50:04.839497 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0812 09:50:04.839574 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.61:39371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260812 09:50:04.840507 19760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:50:04.840233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.61:39371 (local address 127.0.0.1:47156)
0812 09:50:04.840351 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0812 09:50:04.840407 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.61:39371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260812 09:50:04.840659 19758 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.99.61:39371: Network error: Client connection negotiation failed: client connection to 127.13.99.61:39371: connect: Connection refused (error 111)
I20260812 09:50:04.908824 19758 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.99.61" port: 39371 } attempt: 2
I20260812 09:50:04.909653 19760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:50:04.909355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.61:39371 (local address 127.0.0.1:47164)
0812 09:50:04.909507 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0812 09:50:04.909575 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.61:39371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260812 09:50:04.910475 19760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:50:04.910240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.61:39371 (local address 127.0.0.1:47180)
0812 09:50:04.910359 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0812 09:50:04.910414 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.61:39371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260812 09:50:04.910665 19758 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.99.61:39371: Network error: Client connection negotiation failed: client connection to 127.13.99.61:39371: connect: Connection refused (error 111)
W20260812 09:50:04.920269 19771 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 09:50:04.920588 19771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:50:04.920670 19771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:50:04.920746 19771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:50:04.924894 19771 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 09:50:04.924960 19771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:50:04.924980 19771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:50:04.924998 19771 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 09:50:04.925051 19771 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 09:50:04.931092 19771 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:45301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786527905902985-13708-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786527905902985-13708-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.13.99.62:45545,127.13.99.61:39371,127.13.99.60:37813
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.61.keytab
--principal=kudu/127.13.99.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.61:39371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786527905902985-13708-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.13.99.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19771
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786527905902985-13708-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:50:04.932282 19771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:50:04.933538 19771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:50:04.942727 19786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:50:04.942849 19792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:50:04.942894 19771 server_base.cc:1034] running on GCE node
W20260812 09:50:04.942725 19790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:50:04.943504 19771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:50:04.944680 19771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:50:04.945868 19771 hybrid_clock.cc:648] HybridClock initialized: now 1786528204945856 us; error 26 us; skew 500 ppm
Aug 12 09:50:04 dist-test-slave-jq4t krb5kdc[19629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528204, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:50:04.948721 19771 init.cc:377] Logged in from keytab as kudu/127.13.99.61@KRBTEST.COM (short username kudu)
I20260812 09:50:04.949646 19771 webserver.cc:460] Webserver started at http://127.13.99.61:43431/ using document root <none> and password file <none>
I20260812 09:50:04.950227 19771 fs_manager.cc:359] Metadata directory not provided
I20260812 09:50:04.950321 19771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:50:04.950567 19771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:50:04.952731 19771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786527905902985-13708-0/minicluster-data/master-1/data/instance:
uuid: "5da53722159d430da9d34d90b68294ad"
format_stamp: "Formatted at 2026-08-12 09:50:04 on dist-test-slave-jq4t"
server_key: "990ddf1eb66b39ad6e71d556258860d4"
server_key_iv: "ad2308884a8be9c789283621081e4c6e"
server_key_version: "encryptionkey@0"
I20260812 09:50:04.953315 19771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786527905902985-13708-0/minicluster-data/master-1/wal/instance:
uuid: "5da53722159d430da9d34d90b68294ad"
format_stamp: "Formatted at 2026-08-12 09:50:04 on dist-test-slave-jq4t"
server_key: "990ddf1eb66b39ad6e71d556258860d4"
server_key_iv: "ad2308884a8be9c789283621081e4c6e"
server_key_version: "encryptionkey@0"
I20260812 09:50:04.957011 19771 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 09:50:04.959367 19802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:50:04.960616 19771 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 09:50:04.960754 19771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786527905902985-13708-0/minicluster-data/master-1/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786527905902985-13708-0/minicluster-data/master-1/wal
uuid: "5da53722159d430da9d34d90b68294ad"
format_stamp: "Formatted at 2026-08-12 09:50:04 on dist-test-slave-jq4t"
server_key: "990ddf1eb66b39ad6e71d556258860d4"
server_key_iv: "ad2308884a8be9c789283621081e4c6e"
server_key_version: "encryptionkey@0"
I20260812 09:50:04.960866 19771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786527905902985-13708-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786527905902985-13708-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786527905902985-13708-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:50:04.976377 19771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:50:04.979465 19771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:50:04.979679 19771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:50:04.987013 19771 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.61:39371
I20260812 09:50:04.987298 19878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.61:39371 every 8 connection(s)
I20260812 09:50:04.988085 19771 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786527905902985-13708-0/minicluster-data/master-1/data/info.pb
I20260812 09:50:04.990845 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 19771
I20260812 09:50:04.990953 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786527905902985-13708-0/minicluster-data/master-1/wal/instance
I20260812 09:50:04.991027 19886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:50:04.991308 13708 external_mini_cluster.cc:1428] Setting key b327f5349c411387445bff7c0fa24afe
I20260812 09:50:04.995898 19886 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.99.62" port: 45545 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.13.99.60@KRBTEST.COM; defaulting to no policy
I20260812 09:50:05.001855 19758 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.99.61" port: 39371 } attempt: 3
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.99.60@KRBTEST.COM" created.
Aug 12 09:50:05 dist-test-slave-jq4t krb5kdc[19629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528204, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.61@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
Aug 12 09:50:05 dist-test-slave-jq4t krb5kdc[19629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528204, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.62@KRBTEST.COM for kudu/127.13.99.61@KRBTEST.COM
I20260812 09:50:05.013828 19761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:50:04.998182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43246 (local address 127.13.99.62:45545)
0812 09:50:04.998337 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0812 09:50:04.998341 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:50:04.999496 (+ 1155us) server_negotiation.cc:407] Connection header received
0812 09:50:05.000518 (+ 1022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:50:05.000522 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:50:05.000615 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:50:05.000719 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:50:05.002074 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:50:05.002826 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:50:05.004654 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:50:05.005001 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:50:05.005125 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:50:05.008134 (+ 3009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:50:05.008138 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:50:05.008141 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:50:05.008172 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:50:05.010940 (+ 2768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:50:05.011733 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:50:05.011736 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:50:05.011738 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:50:05.011889 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:50:05.012340 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:50:05.012343 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:50:05.012345 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:50:05.012628 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:50:05.012707 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0812 09:50:05.013493 (+ 786us) server_negotiation.cc:299] Negotiation successful
0812 09:50:05.013624 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":50,"spinlock_wait_cycles":2944}
I20260812 09:50:05.014183 19760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:50:05.002328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.61:39371 (local address 127.0.0.1:47182)
0812 09:50:05.002517 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0812 09:50:05.002534 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 09:50:05.002633 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:50:05.003217 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:50:05.003220 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:50:05.003414 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:50:05.003539 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:50:05.003557 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:50:05.004831 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:50:05.004837 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:50:05.005770 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:50:05.005786 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:50:05.006609 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:50:05.006612 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:50:05.006730 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:50:05.007264 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:50:05.007280 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:50:05.008870 (+ 1590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:50:05.011725 (+ 2855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:50:05.011730 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:50:05.011732 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:50:05.012025 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:50:05.012357 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:50:05.012363 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:50:05.012368 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:50:05.012500 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:50:05.013348 (+ 848us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:50:05.013352 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:50:05.013469 (+ 117us) client_negotiation.cc:769] Sending connection context
0812 09:50:05.013588 (+ 119us) client_negotiation.cc:240] Negotiation successful
0812 09:50:05.013733 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114}
I20260812 09:50:05.014780 19898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:50:05.002447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47182 (local address 127.13.99.61:39371)
0812 09:50:05.002771 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0812 09:50:05.002782 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 09:50:05.002807 (+ 25us) server_negotiation.cc:407] Connection header received
0812 09:50:05.002853 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:50:05.002860 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:50:05.003052 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:50:05.003182 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:50:05.003676 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:50:05.004660 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:50:05.005938 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:50:05.006469 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:50:05.006594 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:50:05.009043 (+ 2449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:50:05.009053 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:50:05.009070 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:50:05.009172 (+ 102us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:50:05.011596 (+ 2424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:50:05.012153 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:50:05.012158 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:50:05.012160 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:50:05.012246 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:50:05.012606 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:50:05.012610 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:50:05.012612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:50:05.013211 (+ 599us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:50:05.013347 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0812 09:50:05.013971 (+ 624us) server_negotiation.cc:299] Negotiation successful
0812 09:50:05.014152 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":228,"spinlock_wait_cycles":41344,"thread_start_us":137,"threads_started":1}
I20260812 09:50:05.014798 19897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:50:04.998724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:45545 (local address 127.0.0.1:43246)
0812 09:50:04.999318 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0812 09:50:04.999363 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 09:50:05.000265 (+ 902us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:50:05.000918 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:50:05.000925 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:50:05.001275 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:50:05.001835 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:50:05.001857 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:50:05.002974 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:50:05.002977 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:50:05.004520 (+ 1543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:50:05.004535 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:50:05.005168 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:50:05.005172 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:50:05.005362 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:50:05.005989 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:50:05.006008 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:50:05.007991 (+ 1983us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:50:05.011084 (+ 3093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:50:05.011093 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:50:05.011102 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:50:05.011596 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:50:05.011993 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:50:05.011996 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:50:05.011998 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:50:05.012197 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:50:05.012792 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:50:05.012798 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:50:05.013204 (+ 406us) client_negotiation.cc:769] Sending connection context
0812 09:50:05.013427 (+ 223us) client_negotiation.cc:240] Negotiation successful
0812 09:50:05.013753 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"mutex_wait_us":50,"spinlock_wait_cycles":4736,"thread_start_us":169,"threads_started":1}
I20260812 09:50:05.016093 19886 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.99.61" port: 39371 } has no permanent_uuid. Determining permanent_uuid...
I20260812 09:50:05.017313 19758 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.99.60" port: 37813 } has no permanent_uuid. Determining permanent_uuid...
I20260812 09:50:05.017988 19760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:50:05.017766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.60:37813 (local address 127.0.0.1:37020)
0812 09:50:05.017876 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0812 09:50:05.017931 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.60:37813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260812 09:50:05.018731 19760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:50:05.018452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.60:37813 (local address 127.0.0.1:37030)
0812 09:50:05.018630 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0812 09:50:05.018679 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.60:37813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260812 09:50:05.018926 19758 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.99.60:37813: Network error: Client connection negotiation failed: client connection to 127.13.99.60:37813: connect: Connection refused (error 111)
Aug 12 09:50:05 dist-test-slave-jq4t krb5kdc[19629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528204, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.61@KRBTEST.COM for kudu/127.13.99.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.99.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.60.keytab.
Entry for principal kudu/127.13.99.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.60.keytab.
I20260812 09:50:05.024174 13708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
/tmp/dist-test-taskJsh0qK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786527905902985-13708-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786527905902985-13708-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786527905902985-13708-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786527905902985-13708-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.60:37813
--webserver_interface=127.13.99.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.60.keytab
--principal=kudu/127.13.99.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.99.20:45301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.99.62:45545,127.13.99.61:39371,127.13.99.60:37813
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786527905902985-13708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786527905902985-13708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786527905902985-13708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 09:50:05.025895 19898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:50:05.016991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47184 (local address 127.13.99.61:39371)
0812 09:50:05.017154 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0812 09:50:05.017157 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 09:50:05.017170 (+ 13us) server_negotiation.cc:407] Connection header received
0812 09:50:05.017201 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:50:05.017203 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:50:05.017236 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:50:05.017348 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:50:05.018327 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:50:05.019161 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:50:05.020034 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:50:05.020274 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:50:05.020362 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:50:05.022533 (+ 2171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:50:05.022536 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:50:05.022538 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:50:05.022587 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:50:05.024018 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:50:05.024601 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:50:05.024604 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:50:05.024606 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:50:05.024679 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:50:05.025004 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:50:05.025006 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:50:05.025007 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:50:05.025167 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:50:05.025242 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0812 09:50:05.025576 (+ 334us) server_negotiation.cc:299] Negotiation successful
0812 09:50:05.025760 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":77,"server-negotiator.queue_time_us":73}
I20260812 09:50:05.026000 19897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:50:05.016704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.61:39371 (local address 127.0.0.1:47184)
0812 09:50:05.016816 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0812 09:50:05.016831 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 09:50:05.016903 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:50:05.017429 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:50:05.017432 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:50:05.017605 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:50:05.017714 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:50:05.017720 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:50:05.019261 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:50:05.019263 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:50:05.019861 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:50:05.019871 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:50:05.020374 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:50:05.020377 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:50:05.020459 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:50:05.020969 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:50:05.020980 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:50:05.022377 (+ 1397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:50:05.024133 (+ 1756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:50:05.024136 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:50:05.024138 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:50:05.024423 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:50:05.024775 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:50:05.024781 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:50:05.024786 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:50:05.024893 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:50:05.025261 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:50:05.025267 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:50:05.025397 (+ 130us) client_negotiation.cc:769] Sending connection context
0812 09:50:05.025515 (+ 118us) client_negotiation.cc:240] Negotiation successful
0812 09:50:05.025682 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260812 09:50:05.026724 19886 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.99.60" port: 37813 } has no permanent_uuid. Determining permanent_uuid...
I20260812 09:50:05.027424 19897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:50:05.027176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.60:37813 (local address 127.0.0.1:37044)
0812 09:50:05.027289 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0812 09:50:05.027375 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.60:37813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260812 09:50:05.027648 19803 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.99.60:37813: connect: Connection refused (error 111)
I20260812 09:50:05.028563 19897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:50:05.028304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.60:37813 (local address 127.0.0.1:37056)
0812 09:50:05.028416 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0812 09:50:05.028466 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.60:37813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260812 09:50:05.028728 19886 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.99.60:37813: Network error: Client connection negotiation failed: client connection to 127.13.99.60:37813: connect: Connection refused (error 111)
I20260812 09:50:05.050088 19758 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.99.60" port: 37813 } attempt: 1
I20260812 09:50:05.050957 19760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:50:05.050654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.60:37813 (local address 127.0.0.1:37066)
0812 09:50:05.050816 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0812 09:50:05.050880 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.60:37813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260812 09:50:05.051654 19760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:50:05.051446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.60:37813 (local address 127.0.0.1:37072)
0812 09:50:05.051554 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0812 09:50:05.051604 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.60:37813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260812 09:50:05.051783 19758 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.99.60:37813: Network error: Client connection negotiation failed: client connection to 127.13.99.60:37813: connect: Connection refused (error 111)
I20260812 09:50:05.077977 19886 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.99.60" port: 37813 } attempt: 1
I20260812 09:50:05.078938 19897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:50:05.078560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.60:37813 (local address 127.0.0.1:37076)
0812 09:50:05.078760 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0812 09:50:05.078840 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.60:37813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260812 09:50:05.080086 19897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:50:05.079825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.60:37813 (local address 127.0.0.1:37092)
0812 09:50:05.079951 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0812 09:50:05.080015 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.60:37813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260812 09:50:05.080242 19886 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.99.60:37813: Network error: Client connection negotiation failed: client connection to 127.13.99.60:37813: connect: Connection refused (error 111)
I20260812 09:50:05.126947 19758 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.99.60" port: 37813 } attempt: 2
I20260812 09:50:05.127918 19760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:50:05.127546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.60:37813 (local address 127.0.0.1:37102)
0812 09:50:05.127762 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0812 09:50:05.127829 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.60:37813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260812 09:50:05.128762 19760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:50:05.128467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.60:37813 (local address 127.0.0.1:37116)
0812 09:50:05.128635 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0812 09:50:05.128694 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.60:37813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
W20260812 09:50:05.128943 19758 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.99.60:37813: Network error: Client connection negotiation failed: client connection to 127.13.99.60:37813: connect: Connection refused (error 111)
W20260812 09:50:05.139530 19901 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 09:50:05.139763 19901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:50:05.139833 19901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:50:05.139938 19901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:50:05.143932 19901 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 09:50:05.143982 19901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 09:50:05.144001 19901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:50:05.144060 19901 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 09:50:05.144096 19901 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 09:50:05.148437 19886 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.99.60" port: 37813 } attempt: 2
I20260812 09:50:05.149271 19897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:50:05.148953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.60:37813 (local address 127.0.0.1:37128)
0812 09:50:05.149119 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0812 09:50:05.149183 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.60:37813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260812 09:50:05.149493 19901 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.99.20:45301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786527905902985-13708-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786527905902985-13708-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.13.99.62:45545,127.13.99.61:39371,127.13.99.60:37813
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786527905902985-13708-0/krb5kdc/kudu_127.13.99.60.keytab
--principal=kudu/127.13.99.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.99.60:37813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786527905902985-13708-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.13.99.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19901
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786527905902985-13708-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:50:05.150081 19897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:50:05.149811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.60:37813 (local address 127.0.0.1:37136)
0812 09:50:05.149952 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0812 09:50:05.150016 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.99.60:37813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260812 09:50:05.150272 19886 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.99.60:37813: Network error: Client connection negotiation failed: client connection to 127.13.99.60:37813: connect: Connection refused (error 111)
I20260812 09:50:05.150704 19901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:50:05.151729 19901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 09:50:05.160559 19901 server_base.cc:1034] running on GCE node
W20260812 09:50:05.160498 19911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:50:05.160529 19922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:50:05.160502 19913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:50:05.161427 19901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:50:05.162503 19901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:50:05.163663 19901 hybrid_clock.cc:648] HybridClock initialized: now 1786528205163643 us; error 30 us; skew 500 ppm
Aug 12 09:50:05 dist-test-slave-jq4t krb5kdc[19629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528205, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 09:50:05.166529 19901 init.cc:377] Logged in from keytab as kudu/127.13.99.60@KRBTEST.COM (short username kudu)
I20260812 09:50:05.167431 19901 webserver.cc:460] Webserver started at http://127.13.99.60:42301/ using document root <none> and password file <none>
I20260812 09:50:05.168013 19901 fs_manager.cc:359] Metadata directory not provided
I20260812 09:50:05.168087 19901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:50:05.168321 19901 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:50:05.170444 19901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786527905902985-13708-0/minicluster-data/master-2/data/instance:
uuid: "692206eec4d44e4a9360c0d4cf1b97b3"
format_stamp: "Formatted at 2026-08-12 09:50:05 on dist-test-slave-jq4t"
server_key: "0a804c52f3bb201b47a3da334087baf9"
server_key_iv: "311d62f81a1c705022eda52a7fe81086"
server_key_version: "encryptionkey@0"
I20260812 09:50:05.171036 19901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786527905902985-13708-0/minicluster-data/master-2/wal/instance:
uuid: "692206eec4d44e4a9360c0d4cf1b97b3"
format_stamp: "Formatted at 2026-08-12 09:50:05 on dist-test-slave-jq4t"
server_key: "0a804c52f3bb201b47a3da334087baf9"
server_key_iv: "311d62f81a1c705022eda52a7fe81086"
server_key_version: "encryptionkey@0"
I20260812 09:50:05.174759 19901 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260812 09:50:05.177171 19931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:50:05.178325 19901 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260812 09:50:05.178481 19901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786527905902985-13708-0/minicluster-data/master-2/data,/tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786527905902985-13708-0/minicluster-data/master-2/wal
uuid: "692206eec4d44e4a9360c0d4cf1b97b3"
format_stamp: "Formatted at 2026-08-12 09:50:05 on dist-test-slave-jq4t"
server_key: "0a804c52f3bb201b47a3da334087baf9"
server_key_iv: "311d62f81a1c705022eda52a7fe81086"
server_key_version: "encryptionkey@0"
I20260812 09:50:05.178598 19901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786527905902985-13708-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786527905902985-13708-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786527905902985-13708-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:50:05.211484 19901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:50:05.215047 19901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:50:05.215291 19901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:50:05.222714 19901 rpc_server.cc:307] RPC server started. Bound to: 127.13.99.60:37813
I20260812 09:50:05.223024 20020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.99.60:37813 every 8 connection(s)
I20260812 09:50:05.223742 19901 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786527905902985-13708-0/minicluster-data/master-2/data/info.pb
I20260812 09:50:05.226802 20021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:50:05.228160 19758 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.99.60" port: 37813 } attempt: 3
I20260812 09:50:05.230939 20021 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.99.62" port: 45545 } has no permanent_uuid. Determining permanent_uuid...
I20260812 09:50:05.232601 13708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu as pid 19901
I20260812 09:50:05.232723 13708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJsh0qK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786527905902985-13708-0/minicluster-data/master-2/wal/instance
I20260812 09:50:05.233088 13708 external_mini_cluster.cc:1428] Setting key 20aa6678d9910a316d89f0196aad90d3
Aug 12 09:50:05 dist-test-slave-jq4t krb5kdc[19629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528204, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.62@KRBTEST.COM for kudu/127.13.99.60@KRBTEST.COM
Aug 12 09:50:05 dist-test-slave-jq4t krb5kdc[19629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528205, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.60@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
Aug 12 09:50:05 dist-test-slave-jq4t krb5kdc[19629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528204, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.99.60@KRBTEST.COM
I20260812 09:50:05.241467 19886 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.99.60" port: 37813 } attempt: 3
I20260812 09:50:05.241927 19760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:50:05.228771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.60:37813 (local address 127.0.0.1:37150)
0812 09:50:05.228909 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0812 09:50:05.228925 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 09:50:05.229018 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:50:05.231171 (+ 2153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:50:05.231174 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:50:05.231391 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:50:05.231540 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:50:05.231565 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:50:05.233357 (+ 1792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:50:05.233361 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:50:05.234204 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:50:05.234222 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:50:05.235106 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:50:05.235110 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:50:05.235196 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:50:05.235773 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:50:05.235789 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:50:05.237375 (+ 1586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:50:05.239706 (+ 2331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:50:05.239710 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:50:05.239712 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:50:05.240029 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:50:05.240423 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:50:05.240429 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:50:05.240433 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:50:05.240571 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:50:05.241308 (+ 737us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:50:05.241311 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:50:05.241463 (+ 152us) client_negotiation.cc:769] Sending connection context
0812 09:50:05.241586 (+ 123us) client_negotiation.cc:240] Negotiation successful
0812 09:50:05.241711 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260812 09:50:05.243813 20023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:50:05.229371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37150 (local address 127.13.99.60:37813)
0812 09:50:05.229800 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0812 09:50:05.229819 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0812 09:50:05.229865 (+ 46us) server_negotiation.cc:407] Connection header received
0812 09:50:05.230499 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:50:05.230517 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:50:05.230864 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:50:05.231229 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:50:05.231742 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:50:05.233200 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:50:05.234306 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:50:05.234982 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:50:05.235152 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:50:05.237545 (+ 2393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:50:05.237558 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:50:05.237569 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:50:05.237591 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:50:05.239593 (+ 2002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:50:05.240169 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:50:05.240175 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:50:05.240181 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:50:05.240324 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:50:05.240684 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:50:05.240689 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:50:05.240690 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:50:05.241196 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:50:05.241325 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0812 09:50:05.242277 (+ 952us) server_negotiation.cc:299] Negotiation successful
0812 09:50:05.242577 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":136,"threads_started":1}
I20260812 09:50:05.244036 19761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:50:05.232681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43252 (local address 127.13.99.62:45545)
0812 09:50:05.232802 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0812 09:50:05.232808 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 09:50:05.233357 (+ 549us) server_negotiation.cc:407] Connection header received
0812 09:50:05.233526 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:50:05.233531 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:50:05.233603 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:50:05.233684 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:50:05.234647 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:50:05.235410 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:50:05.236384 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:50:05.236688 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:50:05.236801 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:50:05.239663 (+ 2862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:50:05.239668 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:50:05.239670 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:50:05.239696 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:50:05.242032 (+ 2336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:50:05.242578 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:50:05.242581 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:50:05.242583 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:50:05.242664 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:50:05.242982 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:50:05.242989 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:50:05.242992 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:50:05.243309 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:50:05.243421 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0812 09:50:05.243764 (+ 343us) server_negotiation.cc:299] Negotiation successful
0812 09:50:05.243878 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260812 09:50:05.244081 20026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:50:05.232645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.62:45545 (local address 127.0.0.1:43252)
0812 09:50:05.233216 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0812 09:50:05.233270 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0812 09:50:05.233406 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:50:05.233863 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:50:05.233869 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:50:05.234356 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:50:05.234495 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:50:05.234501 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:50:05.235539 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:50:05.235543 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:50:05.236248 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:50:05.236258 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:50:05.236846 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:50:05.236851 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:50:05.236963 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:50:05.237515 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:50:05.237534 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:50:05.239496 (+ 1962us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:50:05.242153 (+ 2657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:50:05.242159 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:50:05.242164 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:50:05.242455 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:50:05.242775 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:50:05.242778 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:50:05.242780 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:50:05.242876 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:50:05.243423 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:50:05.243431 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:50:05.243557 (+ 126us) client_negotiation.cc:769] Sending connection context
0812 09:50:05.243703 (+ 146us) client_negotiation.cc:240] Negotiation successful
0812 09:50:05.243878 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"mutex_wait_us":1,"thread_start_us":100,"threads_started":1}
I20260812 09:50:05.245891 20021 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.99.61" port: 39371 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 09:50:05 dist-test-slave-jq4t krb5kdc[19629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528204, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.61@KRBTEST.COM for kudu/127.13.99.60@KRBTEST.COM
I20260812 09:50:05.248631 20028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:50:05.233747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37164 (local address 127.13.99.60:37813)
0812 09:50:05.234138 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0812 09:50:05.234147 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 09:50:05.234254 (+ 107us) server_negotiation.cc:407] Connection header received
0812 09:50:05.234377 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:50:05.234380 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:50:05.234430 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:50:05.234528 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:50:05.235449 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:50:05.236285 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:50:05.237506 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:50:05.237895 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:50:05.238003 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:50:05.240936 (+ 2933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:50:05.240940 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:50:05.240942 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:50:05.240963 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:50:05.245949 (+ 4986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:50:05.246467 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:50:05.246470 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:50:05.246471 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:50:05.246574 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:50:05.247175 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:50:05.247178 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:50:05.247179 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:50:05.247393 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:50:05.248234 (+ 841us) server_negotiation.cc:1035] Waiting for connection context
0812 09:50:05.248310 (+ 76us) server_negotiation.cc:299] Negotiation successful
0812 09:50:05.248412 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":330,"thread_start_us":151,"threads_started":1}
I20260812 09:50:05.248864 19758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bb031a243384595910e3baae543a5ac: Bootstrap starting.
I20260812 09:50:05.252007 19758 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7bb031a243384595910e3baae543a5ac: Neither blocks nor log segments found. Creating new log.
Aug 12 09:50:05 dist-test-slave-jq4t krb5kdc[19629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528205, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.60@KRBTEST.COM for kudu/127.13.99.61@KRBTEST.COM
I20260812 09:50:05.252641 19897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:50:05.241954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.60:37813 (local address 127.0.0.1:37168)
0812 09:50:05.242182 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0812 09:50:05.242202 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 09:50:05.242271 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:50:05.242794 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:50:05.242798 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:50:05.243011 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:50:05.243123 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:50:05.243131 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:50:05.244138 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:50:05.244141 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:50:05.244946 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:50:05.244962 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:50:05.246402 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:50:05.246406 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:50:05.246494 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:50:05.246968 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:50:05.246985 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:50:05.248393 (+ 1408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:50:05.250964 (+ 2571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:50:05.250967 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:50:05.250969 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:50:05.251204 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:50:05.251525 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:50:05.251528 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:50:05.251530 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:50:05.251619 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:50:05.252103 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:50:05.252106 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:50:05.252207 (+ 101us) client_negotiation.cc:769] Sending connection context
0812 09:50:05.252301 (+ 94us) client_negotiation.cc:240] Negotiation successful
0812 09:50:05.252437 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151}
I20260812 09:50:05.253122 19758 log.cc:824] T 00000000000000000000000000000000 P 7bb031a243384595910e3baae543a5ac: Log is configured to *not* fsync() on all Append() calls
I20260812 09:50:05.253454 20029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:50:05.241977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37168 (local address 127.13.99.60:37813)
0812 09:50:05.242582 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0812 09:50:05.242589 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 09:50:05.242599 (+ 10us) server_negotiation.cc:407] Connection header received
0812 09:50:05.242626 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:50:05.242628 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:50:05.242659 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:50:05.242760 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:50:05.243254 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:50:05.244001 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:50:05.245955 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:50:05.246263 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:50:05.246723 (+ 460us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:50:05.248582 (+ 1859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:50:05.248586 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:50:05.248589 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:50:05.248612 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:50:05.250769 (+ 2157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:50:05.251331 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:50:05.251334 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:50:05.251336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:50:05.251420 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:50:05.251732 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:50:05.251735 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:50:05.251737 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:50:05.251970 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:50:05.253109 (+ 1139us) server_negotiation.cc:1035] Waiting for connection context
0812 09:50:05.253190 (+ 81us) server_negotiation.cc:299] Negotiation successful
0812 09:50:05.253309 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":91,"threads_started":1}
I20260812 09:50:05.255395 19758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bb031a243384595910e3baae543a5ac: No bootstrap required, opened a new log
I20260812 09:50:05.256654 19886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5da53722159d430da9d34d90b68294ad: Bootstrap starting.
I20260812 09:50:05.256991 19898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:50:05.246632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47194 (local address 127.13.99.61:39371)
0812 09:50:05.246753 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0812 09:50:05.246758 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:50:05.246919 (+ 161us) server_negotiation.cc:407] Connection header received
0812 09:50:05.247119 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:50:05.247125 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:50:05.247184 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:50:05.247238 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:50:05.247800 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:50:05.248539 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:50:05.249869 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:50:05.250250 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:50:05.250335 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:50:05.252971 (+ 2636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:50:05.252975 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:50:05.252978 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:50:05.253002 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:50:05.254762 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:50:05.255395 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:50:05.255402 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:50:05.255410 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:50:05.255507 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:50:05.255851 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:50:05.255857 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:50:05.255861 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:50:05.256173 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:50:05.256261 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0812 09:50:05.256706 (+ 445us) server_negotiation.cc:299] Negotiation successful
0812 09:50:05.256820 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
Aug 12 09:50:05 dist-test-slave-jq4t krb5kdc[19629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528204, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.99.62@KRBTEST.COM
I20260812 09:50:05.257500 20026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:50:05.246396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.61:39371 (local address 127.0.0.1:47194)
0812 09:50:05.246858 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0812 09:50:05.246872 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 09:50:05.246985 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:50:05.247398 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:50:05.247401 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:50:05.247566 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:50:05.247669 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:50:05.247676 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:50:05.248686 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:50:05.248691 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:50:05.249731 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:50:05.249742 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:50:05.250486 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:50:05.250489 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:50:05.250939 (+ 450us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:50:05.251416 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:50:05.251434 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:50:05.252830 (+ 1396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:50:05.254995 (+ 2165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:50:05.254998 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:50:05.255000 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:50:05.255275 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:50:05.255638 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:50:05.255641 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:50:05.255642 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:50:05.255731 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:50:05.256351 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:50:05.256355 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:50:05.256440 (+ 85us) client_negotiation.cc:769] Sending connection context
0812 09:50:05.257170 (+ 730us) client_negotiation.cc:240] Negotiation successful
0812 09:50:05.257315 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":389}
I20260812 09:50:05.258201 20021 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.99.60" port: 37813 } has no permanent_uuid. Determining permanent_uuid...
I20260812 09:50:05.259555 19758 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7bb031a243384595910e3baae543a5ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb031a243384595910e3baae543a5ac" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 45545 } } peers { permanent_uuid: "5da53722159d430da9d34d90b68294ad" member_type: VOTER last_known_addr { host: "127.13.99.61" port: 39371 } } peers { permanent_uuid: "692206eec4d44e4a9360c0d4cf1b97b3" member_type: VOTER last_known_addr { host: "127.13.99.60" port: 37813 } }
I20260812 09:50:05.259809 19758 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7bb031a243384595910e3baae543a5ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bb031a243384595910e3baae543a5ac, State: Initialized, Role: FOLLOWER
I20260812 09:50:05.260195 19758 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7bb031a243384595910e3baae543a5ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb031a243384595910e3baae543a5ac" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 45545 } } peers { permanent_uuid: "5da53722159d430da9d34d90b68294ad" member_type: VOTER last_known_addr { host: "127.13.99.61" port: 39371 } } peers { permanent_uuid: "692206eec4d44e4a9360c0d4cf1b97b3" member_type: VOTER last_known_addr { host: "127.13.99.60" port: 37813 } }
I20260812 09:50:05.261067 19886 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5da53722159d430da9d34d90b68294ad: Neither blocks nor log segments found. Creating new log.
I20260812 09:50:05.261663 19758 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7bb031a243384595910e3baae543a5ac [sys.catalog]: configured and running, proceeding with master startup.
I20260812 09:50:05.261873 19761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:50:05.250522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43268 (local address 127.13.99.62:45545)
0812 09:50:05.250636 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0812 09:50:05.250640 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:50:05.250652 (+ 12us) server_negotiation.cc:407] Connection header received
0812 09:50:05.250691 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:50:05.250693 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:50:05.250725 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:50:05.250830 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:50:05.251913 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:50:05.252730 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:50:05.254411 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:50:05.254780 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:50:05.254875 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:50:05.257750 (+ 2875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:50:05.257753 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:50:05.257756 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:50:05.257780 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:50:05.259563 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:50:05.260068 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:50:05.260074 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:50:05.260077 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:50:05.260167 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:50:05.260464 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:50:05.260467 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:50:05.260469 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:50:05.260698 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:50:05.260898 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0812 09:50:05.261590 (+ 692us) server_negotiation.cc:299] Negotiation successful
0812 09:50:05.261717 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260812 09:50:05.262511 19886 log.cc:824] T 00000000000000000000000000000000 P 5da53722159d430da9d34d90b68294ad: Log is configured to *not* fsync() on all Append() calls
Aug 12 09:50:05 dist-test-slave-jq4t krb5kdc[19629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528205, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.99.60@KRBTEST.COM for kudu/127.13.99.60@KRBTEST.COM
I20260812 09:50:05.265025 19886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5da53722159d430da9d34d90b68294ad: No bootstrap required, opened a new log
I20260812 09:50:05.265249 20033 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bb031a243384595910e3baae543a5ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb031a243384595910e3baae543a5ac" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 45545 } } peers { permanent_uuid: "5da53722159d430da9d34d90b68294ad" member_type: VOTER last_known_addr { host: "127.13.99.61" port: 39371 } } peers { permanent_uuid: "692206eec4d44e4a9360c0d4cf1b97b3" member_type: VOTER last_known_addr { host: "127.13.99.60" port: 37813 } } }
I20260812 09:50:05.265388 20033 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bb031a243384595910e3baae543a5ac [sys.catalog]: This master's current role is: FOLLOWER
I20260812 09:50:05.268378 20026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:50:05.258670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.99.60:37813 (local address 127.0.0.1:37172)
0812 09:50:05.259137 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0812 09:50:05.259151 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 09:50:05.259225 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 09:50:05.259615 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 09:50:05.259619 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 09:50:05.259792 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0812 09:50:05.259893 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:50:05.259902 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:50:05.260912 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:50:05.260918 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:50:05.261649 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 09:50:05.261662 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 09:50:05.262311 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 09:50:05.262314 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 09:50:05.262400 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:50:05.262834 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 09:50:05.262851 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 09:50:05.264440 (+ 1589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 09:50:05.266838 (+ 2398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:50:05.266840 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:50:05.266842 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:50:05.267095 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:50:05.267336 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 09:50:05.267338 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 09:50:05.267340 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 09:50:05.267411 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 09:50:05.267811 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 09:50:05.267813 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 09:50:05.267914 (+ 101us) client_negotiation.cc:769] Sending connection context
0812 09:50:05.268025 (+ 111us) client_negotiation.cc:240] Negotiation successful
0812 09:50:05.268189 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":106,"threads_started":1}
I20260812 09:50:05.269019 20029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:50:05.258999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37172 (local address 127.13.99.60:37813)
0812 09:50:05.259364 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0812 09:50:05.259368 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:50:05.259378 (+ 10us) server_negotiation.cc:407] Connection header received
0812 09:50:05.259409 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:50:05.259415 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:50:05.259456 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:50:05.259523 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:50:05.260021 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:50:05.260803 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:50:05.261774 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:50:05.262107 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:50:05.262207 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:50:05.264603 (+ 2396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:50:05.264606 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:50:05.264608 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:50:05.264634 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:50:05.266729 (+ 2095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:50:05.267186 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:50:05.267188 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:50:05.267190 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:50:05.267259 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:50:05.267493 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:50:05.267497 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:50:05.267500 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:50:05.267692 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:50:05.267779 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0812 09:50:05.268700 (+ 921us) server_negotiation.cc:299] Negotiation successful
0812 09:50:05.268832 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298}
Aug 12 09:50:05 dist-test-slave-jq4t krb5kdc[19629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786528204, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.99.61@KRBTEST.COM
I20260812 09:50:05.269889 19886 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5da53722159d430da9d34d90b68294ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb031a243384595910e3baae543a5ac" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 45545 } } peers { permanent_uuid: "5da53722159d430da9d34d90b68294ad" member_type: VOTER last_known_addr { host: "127.13.99.61" port: 39371 } } peers { permanent_uuid: "692206eec4d44e4a9360c0d4cf1b97b3" member_type: VOTER last_known_addr { host: "127.13.99.60" port: 37813 } }
I20260812 09:50:05.270300 19886 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5da53722159d430da9d34d90b68294ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5da53722159d430da9d34d90b68294ad, State: Initialized, Role: FOLLOWER
I20260812 09:50:05.270689 19886 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5da53722159d430da9d34d90b68294ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb031a243384595910e3baae543a5ac" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 45545 } } peers { permanent_uuid: "5da53722159d430da9d34d90b68294ad" member_type: VOTER last_known_addr { host: "127.13.99.61" port: 39371 } } peers { permanent_uuid: "692206eec4d44e4a9360c0d4cf1b97b3" member_type: VOTER last_known_addr { host: "127.13.99.60" port: 37813 } }
I20260812 09:50:05.267639 19649 master_runner.cc:186] Error getting master registration for 127.13.99.61:39371: OK, instance_id { permanent_uuid: "5da53722159d430da9d34d90b68294ad" instance_seqno: 1786528204984954 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 09:50:05.271657 19649 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 09:50:05.272197 19886 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5da53722159d430da9d34d90b68294ad [sys.catalog]: configured and running, proceeding with master startup.
I20260812 09:50:05.272188 20021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 692206eec4d44e4a9360c0d4cf1b97b3: Bootstrap starting.
I20260812 09:50:05.274245 19898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:50:05.263352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47198 (local address 127.13.99.61:39371)
0812 09:50:05.263465 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0812 09:50:05.263472 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 09:50:05.263486 (+ 14us) server_negotiation.cc:407] Connection header received
0812 09:50:05.263641 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:50:05.263644 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:50:05.263697 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:50:05.263785 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:50:05.264660 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:50:05.265647 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:50:05.267100 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:50:05.267476 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:50:05.267586 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:50:05.270487 (+ 2901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:50:05.270491 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:50:05.270493 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:50:05.270517 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:50:05.272263 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:50:05.272772 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:50:05.272776 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:50:05.272778 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:50:05.272888 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:50:05.273232 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:50:05.273235 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:50:05.273237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:50:05.273452 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:50:05.273634 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0812 09:50:05.273992 (+ 358us) server_negotiation.cc:299] Negotiation successful
0812 09:50:05.274106 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260812 09:50:05.275087 20021 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 692206eec4d44e4a9360c0d4cf1b97b3: Neither blocks nor log segments found. Creating new log.
I20260812 09:50:05.275424 20054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5da53722159d430da9d34d90b68294ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb031a243384595910e3baae543a5ac" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 45545 } } peers { permanent_uuid: "5da53722159d430da9d34d90b68294ad" member_type: VOTER last_known_addr { host: "127.13.99.61" port: 39371 } } peers { permanent_uuid: "692206eec4d44e4a9360c0d4cf1b97b3" member_type: VOTER last_known_addr { host: "127.13.99.60" port: 37813 } } }
I20260812 09:50:05.275552 20054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5da53722159d430da9d34d90b68294ad [sys.catalog]: This master's current role is: FOLLOWER
I20260812 09:50:05.275978 20021 log.cc:824] T 00000000000000000000000000000000 P 692206eec4d44e4a9360c0d4cf1b97b3: Log is configured to *not* fsync() on all Append() calls
I20260812 09:50:05.278240 20021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 692206eec4d44e4a9360c0d4cf1b97b3: No bootstrap required, opened a new log
I20260812 09:50:05.278239 13708 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260812 09:50:05.279072 20071 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5da53722159d430da9d34d90b68294ad: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 09:50:05.279160 20071 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 09:50:05.279464 19720 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "7bb031a243384595910e3baae543a5ac"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43268
I20260812 09:50:05.279680 19720 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7bb031a243384595910e3baae543a5ac [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 09:50:05.279775 19720 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7bb031a243384595910e3baae543a5ac [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:50:05.281114 19720 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7bb031a243384595910e3baae543a5ac [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb031a243384595910e3baae543a5ac" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 45545 } } peers { permanent_uuid: "5da53722159d430da9d34d90b68294ad" member_type: VOTER last_known_addr { host: "127.13.99.61" port: 39371 } } peers { permanent_uuid: "692206eec4d44e4a9360c0d4cf1b97b3" member_type: VOTER last_known_addr { host: "127.13.99.60" port: 37813 } }
W20260812 09:50:05.281926 20052 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7bb031a243384595910e3baae543a5ac: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 09:50:05.281984 20052 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 09:50:05.282627 19841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7bb031a243384595910e3baae543a5ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5da53722159d430da9d34d90b68294ad"
I20260812 09:50:05.282859 19841 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5da53722159d430da9d34d90b68294ad [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:50:05.282917 19720 leader_election.cc:288] T 00000000000000000000000000000000 P 7bb031a243384595910e3baae543a5ac [CANDIDATE]: Term 1 election: Requested vote from peers 5da53722159d430da9d34d90b68294ad (127.13.99.61:39371), 692206eec4d44e4a9360c0d4cf1b97b3 (127.13.99.60:37813)
I20260812 09:50:05.279486 19771 master_runner.cc:186] Error getting master registration for 127.13.99.60:37813: OK, instance_id { permanent_uuid: "692206eec4d44e4a9360c0d4cf1b97b3" instance_seqno: 1786528205220659 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 09:50:05.283402 19771 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 09:50:05.284217 19841 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5da53722159d430da9d34d90b68294ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7bb031a243384595910e3baae543a5ac in term 1.
I20260812 09:50:05.284945 19683 leader_election.cc:302] T 00000000000000000000000000000000 P 7bb031a243384595910e3baae543a5ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5da53722159d430da9d34d90b68294ad, 7bb031a243384595910e3baae543a5ac; no voters:
I20260812 09:50:05.285286 20033 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7bb031a243384595910e3baae543a5ac [term 1 FOLLOWER]: Leader election won for term 1
I20260812 09:50:05.285499 20033 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7bb031a243384595910e3baae543a5ac [term 1 LEADER]: Becoming Leader. State: Replica: 7bb031a243384595910e3baae543a5ac, State: Running, Role: LEADER
I20260812 09:50:05.285825 20033 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7bb031a243384595910e3baae543a5ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb031a243384595910e3baae543a5ac" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 45545 } } peers { permanent_uuid: "5da53722159d430da9d34d90b68294ad" member_type: VOTER last_known_addr { host: "127.13.99.61" port: 39371 } } peers { permanent_uuid: "692206eec4d44e4a9360c0d4cf1b97b3" member_type: VOTER last_known_addr { host: "127.13.99.60" port: 37813 } }
I20260812 09:50:05.287755 20033 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bb031a243384595910e3baae543a5ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7bb031a243384595910e3baae543a5ac. Latest consensus state: current_term: 1 leader_uuid: "7bb031a243384595910e3baae543a5ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb031a243384595910e3baae543a5ac" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 45545 } } peers { permanent_uuid: "5da53722159d430da9d34d90b68294ad" member_type: VOTER last_known_addr { host: "127.13.99.61" port: 39371 } } peers { permanent_uuid: "692206eec4d44e4a9360c0d4cf1b97b3" member_type: VOTER last_known_addr { host: "127.13.99.60" port: 37813 } } }
I20260812 09:50:05.287868 20033 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bb031a243384595910e3baae543a5ac [sys.catalog]: This master's current role is: LEADER
I20260812 09:50:05.291258 20021 raft_consensus.cc:348] T 00000000000000000000000000000000 P 692206eec4d44e4a9360c0d4cf1b97b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb031a243384595910e3baae543a5ac" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 45545 } } peers { permanent_uuid: "5da53722159d430da9d34d90b68294ad" member_type: VOTER last_known_addr { host: "127.13.99.61" port: 39371 } } peers { permanent_uuid: "692206eec4d44e4a9360c0d4cf1b97b3" member_type: VOTER last_known_addr { host: "127.13.99.60" port: 37813 } }
I20260812 09:50:05.291662 20021 raft_consensus.cc:729] T 00000000000000000000000000000000 P 692206eec4d44e4a9360c0d4cf1b97b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 692206eec4d44e4a9360c0d4cf1b97b3, State: Initialized, Role: FOLLOWER
I20260812 09:50:05.292202 20021 consensus_queue.cc:260] T 00000000000000000000000000000000 P 692206eec4d44e4a9360c0d4cf1b97b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb031a243384595910e3baae543a5ac" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 45545 } } peers { permanent_uuid: "5da53722159d430da9d34d90b68294ad" member_type: VOTER last_known_addr { host: "127.13.99.61" port: 39371 } } peers { permanent_uuid: "692206eec4d44e4a9360c0d4cf1b97b3" member_type: VOTER last_known_addr { host: "127.13.99.60" port: 37813 } }
I20260812 09:50:05.294790 19977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7bb031a243384595910e3baae543a5ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "692206eec4d44e4a9360c0d4cf1b97b3"
I20260812 09:50:05.295063 19977 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 692206eec4d44e4a9360c0d4cf1b97b3 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:50:05.296588 19977 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 692206eec4d44e4a9360c0d4cf1b97b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7bb031a243384595910e3baae543a5ac in term 1.
I20260812 09:50:05.297228 20021 sys_catalog.cc:564] T 00000000000000000000000000000000 P 692206eec4d44e4a9360c0d4cf1b97b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 09:50:05.298735 19898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:50:05.288183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47428 (local address 127.13.99.61:39371)
0812 09:50:05.288284 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0812 09:50:05.288288 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:50:05.288301 (+ 13us) server_negotiation.cc:407] Connection header received
0812 09:50:05.288337 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:50:05.288339 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:50:05.288377 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:50:05.288448 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:50:05.289469 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:50:05.290026 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:50:05.291502 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:50:05.291851 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:50:05.291955 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:50:05.294203 (+ 2248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:50:05.294207 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:50:05.294210 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:50:05.294237 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:50:05.295988 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:50:05.296591 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:50:05.296595 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:50:05.296599 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:50:05.296692 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:50:05.297407 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:50:05.297410 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:50:05.297412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:50:05.297652 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:50:05.297775 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0812 09:50:05.298440 (+ 665us) server_negotiation.cc:299] Negotiation successful
0812 09:50:05.298582 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260812 09:50:05.300113 20029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:50:05.285779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40990 (local address 127.13.99.60:37813)
0812 09:50:05.285920 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0812 09:50:05.285927 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 09:50:05.286607 (+ 680us) server_negotiation.cc:407] Connection header received
0812 09:50:05.286702 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:50:05.286706 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:50:05.286755 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:50:05.286882 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:50:05.287731 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:50:05.288465 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:50:05.289696 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:50:05.290056 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:50:05.290163 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:50:05.292224 (+ 2061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:50:05.292228 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:50:05.292231 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:50:05.292258 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:50:05.293940 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:50:05.294395 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:50:05.294399 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:50:05.294401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:50:05.294486 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:50:05.294778 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:50:05.294781 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:50:05.294783 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:50:05.297033 (+ 2250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:50:05.297197 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0812 09:50:05.299817 (+ 2620us) server_negotiation.cc:299] Negotiation successful
0812 09:50:05.299938 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260812 09:50:05.300729 20095 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 09:50:05.301819 20095 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 09:50:05.302449 19761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:50:05.286100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51208 (local address 127.13.99.62:45545)
0812 09:50:05.286228 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0812 09:50:05.286232 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:50:05.292722 (+ 6490us) server_negotiation.cc:407] Connection header received
0812 09:50:05.292912 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:50:05.292917 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:50:05.292971 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:50:05.293029 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:50:05.293923 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:50:05.294663 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:50:05.295707 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:50:05.296059 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:50:05.296160 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:50:05.297996 (+ 1836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 09:50:05.298000 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 09:50:05.298002 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 09:50:05.298028 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 09:50:05.300016 (+ 1988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:50:05.301015 (+ 999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:50:05.301021 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:50:05.301025 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:50:05.301118 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 09:50:05.301455 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 09:50:05.301459 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 09:50:05.301461 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 09:50:05.301682 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 09:50:05.301787 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0812 09:50:05.302192 (+ 405us) server_negotiation.cc:299] Negotiation successful
0812 09:50:05.302307 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260812 09:50:05.304080 20096 sys_catalog.cc:455] T 00000000000000000000000000000000 P 692206eec4d44e4a9360c0d4cf1b97b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb031a243384595910e3baae543a5ac" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 45545 } } peers { permanent_uuid: "5da53722159d430da9d34d90b68294ad" member_type: VOTER last_known_addr { host: "127.13.99.61" port: 39371 } } peers { permanent_uuid: "692206eec4d44e4a9360c0d4cf1b97b3" member_type: VOTER last_known_addr { host: "127.13.99.60" port: 37813 } } }
I20260812 09:50:05.304193 20096 sys_catalog.cc:458] T 00000000000000000000000000000000 P 692206eec4d44e4a9360c0d4cf1b97b3 [sys.catalog]: This master's current role is: FOLLOWER
W20260812 09:50:05.306281 20119 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 692206eec4d44e4a9360c0d4cf1b97b3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 09:50:05.306363 20119 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 09:50:05.308118 19977 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 692206eec4d44e4a9360c0d4cf1b97b3 [term 1 FOLLOWER]: Refusing update from remote peer 7bb031a243384595910e3baae543a5ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 09:50:05.308838 20087 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7bb031a243384595910e3baae543a5ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "692206eec4d44e4a9360c0d4cf1b97b3" member_type: VOTER last_known_addr { host: "127.13.99.60" port: 37813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 09:50:05.309827 19841 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5da53722159d430da9d34d90b68294ad [term 1 FOLLOWER]: Refusing update from remote peer 7bb031a243384595910e3baae543a5ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 09:50:05.313302 20087 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7bb031a243384595910e3baae543a5ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "5da53722159d430da9d34d90b68294ad" member_type: VOTER last_known_addr { host: "127.13.99.61" port: 39371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 09:50:05.315008 20096 sys_catalog.cc:455] T 00000000000000000000000000000000 P 692206eec4d44e4a9360c0d4cf1b97b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7bb031a243384595910e3baae543a5ac. Latest consensus state: current_term: 1 leader_uuid: "7bb031a243384595910e3baae543a5ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb031a243384595910e3baae543a5ac" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 45545 } } peers { permanent_uuid: "5da53722159d430da9d34d90b68294ad" member_type: VOTER last_known_addr { host: "127.13.99.61" port: 39371 } } peers { permanent_uuid: "692206eec4d44e4a9360c0d4cf1b97b3" member_type: VOTER last_known_addr { host: "127.13.99.60" port: 37813 } } }
I20260812 09:50:05.315133 20096 sys_catalog.cc:458] T 00000000000000000000000000000000 P 692206eec4d44e4a9360c0d4cf1b97b3 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 09:50:05.317022 20054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5da53722159d430da9d34d90b68294ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7bb031a243384595910e3baae543a5ac. Latest consensus state: current_term: 1 leader_uuid: "7bb031a243384595910e3baae543a5ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb031a243384595910e3baae543a5ac" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 45545 } } peers { permanent_uuid: "5da53722159d430da9d34d90b68294ad" member_type: VOTER last_known_addr { host: "127.13.99.61" port: 39371 } } peers { permanent_uuid: "692206eec4d44e4a9360c0d4cf1b97b3" member_type: VOTER last_known_addr { host: "127.13.99.60" port: 37813 } } }
I20260812 09:50:05.317152 20054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5da53722159d430da9d34d90b68294ad [sys.catalog]: This master's current role is: FOLLOWER
I20260812 09:50:05.319842 20087 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bb031a243384595910e3baae543a5ac [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7bb031a243384595910e3baae543a5ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb031a243384595910e3baae543a5ac" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 45545 } } peers { permanent_uuid: "5da53722159d430da9d34d90b68294ad" member_type: VOTER last_known_addr { host: "127.13.99.61" port: 39371 } } peers { permanent_uuid: "692206eec4d44e4a9360c0d4cf1b97b3" member_type: VOTER last_known_addr { host: "127.13.99.60" port: 37813 } } }
I20260812 09:50:05.319967 20087 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bb031a243384595910e3baae543a5ac [sys.catalog]: This master's current role is: LEADER
I20260812 09:50:05.320104 20120 mvcc.cc:205] Tried to move back new op lower bound from 7317619528935735296 to 7317619528854618112. Current Snapshot: MvccSnapshot[applied={T|T < 7317619528935735296}]
I20260812 09:50:05.321223 20054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5da53722159d430da9d34d90b68294ad [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7bb031a243384595910e3baae543a5ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb031a243384595910e3baae543a5ac" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 45545 } } peers { permanent_uuid: "5da53722159d430da9d34d90b68294ad" member_type: VOTER last_known_addr { host: "127.13.99.61" port: 39371 } } peers { permanent_uuid: "692206eec4d44e4a9360c0d4cf1b97b3" member_type: VOTER last_known_addr { host: "127.13.99.60" port: 37813 } } }
I20260812 09:50:05.321357 20054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5da53722159d430da9d34d90b68294ad [sys.catalog]: This master's current role is: FOLLOWER
I20260812 09:50:05.324290 20125 mvcc.cc:205] Tried to move back new op lower bound from 7317619528935735296 to 7317619528854618112. Current Snapshot: MvccSnapshot[applied={T|T < 7317619528935735296}]
I20260812 09:50:05.330216 20095 catalog_manager.cc:1347] Generated new cluster ID: 4cf7ccd296d645ac8fad2cfda1f43d77
I20260812 09:50:05.330293 20095 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 09:50:05.331315 20033 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bb031a243384595910e3baae543a5ac [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7bb031a243384595910e3baae543a5ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb031a243384595910e3baae543a5ac" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 45545 } } peers { permanent_uuid: "5da53722159d430da9d34d90b68294ad" member_type: VOTER last_known_addr { host: "127.13.99.61" port: 39371 } } peers { permanent_uuid: "692206eec4d44e4a9360c0d4cf1b97b3" member_type: VOTER last_known_addr { host: "127.13.99.60" port: 37813 } } }
I20260812 09:50:05.331422 20033 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bb031a243384595910e3baae543a5ac [sys.catalog]: This master's current role is: LEADER
I20260812 09:50:05.333364 20096 sys_catalog.cc:455] T 00000000000000000000000000000000 P 692206eec4d44e4a9360c0d4cf1b97b3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7bb031a243384595910e3baae543a5ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb031a243384595910e3baae543a5ac" member_type: VOTER last_known_addr { host: "127.13.99.62" port: 45545 } } peers { permanent_uuid: "5da53722159d430da9d34d90b68294ad" member_type: VOTER last_known_addr { host: "127.13.99.61" port: 39371 } } peers { permanent_uuid: "692206eec4d44e4a9360c0d4cf1b97b3" member_type: VOTER last_known_addr { host: "127.13.99.60" port: 37813 } } }
I20260812 09:50:05.333464 20096 sys_catalog.cc:458] T 00000000000000000000000000000000 P 692206eec4d44e4a9360c0d4cf1b97b3 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 09:50:05.336597 20123 mvcc.cc:205] Tried to move back new op lower bound from 7317619528935735296 to 7317619528854618112. Current Snapshot: MvccSnapshot[applied={T|T < 7317619528935735296 or (T in {7317619528935735296})}]
I20260812 09:50:05.348938 20095 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 09:50:05.350204 20095 catalog_manager.cc:1504] Loading token signing keys...
I20260812 09:50:05.370281 20095 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7bb031a243384595910e3baae543a5ac: Generated new TSK 0
I20260812 09:50:05.371188 20095 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 09:50:05.403438 19761 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 09:50:05.401508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51210 (local address 127.13.99.62:45545)
0812 09:50:05.401641 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0812 09:50:05.401647 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 09:50:05.402313 (+ 666us) server_negotiation.cc:407] Connection header received
0812 09:50:05.402519 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:50:05.402522 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:50:05.402573 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:50:05.402627 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:50:05.403287 (+ 660us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51210: BlockingRecv error: recv got EOF from 127.0.0.1:51210 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260812 09:50:05.405900 19898 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 09:50:05.404134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47444 (local address 127.13.99.61:39371)
0812 09:50:05.404253 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0812 09:50:05.404257 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:50:05.404963 (+ 706us) server_negotiation.cc:407] Connection header received
0812 09:50:05.405147 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:50:05.405151 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:50:05.405202 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:50:05.405253 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:50:05.405762 (+ 509us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47444: BlockingRecv error: recv got EOF from 127.0.0.1:47444 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260812 09:50:05.407440 20029 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 09:50:05.404681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40998 (local address 127.13.99.60:37813)
0812 09:50:05.404794 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0812 09:50:05.404799 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:50:05.406480 (+ 1681us) server_negotiation.cc:407] Connection header received
0812 09:50:05.406653 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:50:05.406657 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:50:05.406712 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:50:05.406767 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:50:05.407318 (+ 551us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40998: BlockingRecv error: recv got EOF from 127.0.0.1:40998 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260812 09:50:05.411870 19898 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 09:50:05.410184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47454 (local address 127.13.99.61:39371)
0812 09:50:05.410284 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0812 09:50:05.410288 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:50:05.410692 (+ 404us) server_negotiation.cc:407] Connection header received
0812 09:50:05.411073 (+ 381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:50:05.411077 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:50:05.411167 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:50:05.411236 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:50:05.411767 (+ 531us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47454: BlockingRecv error: recv got EOF from 127.0.0.1:47454 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260812 09:50:05.417117 19898 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 09:50:05.415596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47462 (local address 127.13.99.61:39371)
0812 09:50:05.415695 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0812 09:50:05.415699 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:50:05.416180 (+ 481us) server_negotiation.cc:407] Connection header received
0812 09:50:05.416365 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:50:05.416368 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:50:05.416421 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:50:05.416512 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:50:05.417017 (+ 505us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47462: BlockingRecv error: recv got EOF from 127.0.0.1:47462 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260812 09:50:05.422644 19898 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 09:50:05.421269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47468 (local address 127.13.99.61:39371)
0812 09:50:05.421362 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0812 09:50:05.421366 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:50:05.421798 (+ 432us) server_negotiation.cc:407] Connection header received
0812 09:50:05.421995 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:50:05.421998 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:50:05.422046 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:50:05.422095 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:50:05.422553 (+ 458us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47468: BlockingRecv error: recv got EOF from 127.0.0.1:47468 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260812 09:50:05.430330 19898 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 09:50:05.428839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47476 (local address 127.13.99.61:39371)
0812 09:50:05.428987 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0812 09:50:05.428991 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:50:05.429467 (+ 476us) server_negotiation.cc:407] Connection header received
0812 09:50:05.429655 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:50:05.429659 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:50:05.429706 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:50:05.429754 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:50:05.430235 (+ 481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47476: BlockingRecv error: recv got EOF from 127.0.0.1:47476 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
W20260812 09:50:05.442123 19898 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 09:50:05.440634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47482 (local address 127.13.99.61:39371)
0812 09:50:05.440731 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0812 09:50:05.440735 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:50:05.441218 (+ 483us) server_negotiation.cc:407] Connection header received
0812 09:50:05.441413 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:50:05.441416 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:50:05.441466 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:50:05.441518 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:50:05.442027 (+ 509us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47482: BlockingRecv error: recv got EOF from 127.0.0.1:47482 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260812 09:50:05.462392 19898 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 09:50:05.460690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47486 (local address 127.13.99.61:39371)
0812 09:50:05.460805 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0812 09:50:05.460811 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 09:50:05.461352 (+ 541us) server_negotiation.cc:407] Connection header received
0812 09:50:05.461579 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:50:05.461583 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:50:05.461639 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:50:05.461693 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:50:05.462281 (+ 588us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47486: BlockingRecv error: recv got EOF from 127.0.0.1:47486 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260812 09:50:05.499179 19898 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 09:50:05.497258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47494 (local address 127.13.99.61:39371)
0812 09:50:05.497383 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0812 09:50:05.497388 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:50:05.498158 (+ 770us) server_negotiation.cc:407] Connection header received
0812 09:50:05.498371 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:50:05.498375 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:50:05.498435 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:50:05.498491 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:50:05.499063 (+ 572us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47494: BlockingRecv error: recv got EOF from 127.0.0.1:47494 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260812 09:50:05.567051 19898 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 09:50:05.565923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47504 (local address 127.13.99.61:39371)
0812 09:50:05.566108 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0812 09:50:05.566113 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:50:05.566184 (+ 71us) server_negotiation.cc:407] Connection header received
0812 09:50:05.566322 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:50:05.566325 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:50:05.566377 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:50:05.566451 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:50:05.566916 (+ 465us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47504: BlockingRecv error: recv got EOF from 127.0.0.1:47504 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260812 09:50:05.698706 19898 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 09:50:05.697308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47508 (local address 127.13.99.61:39371)
0812 09:50:05.697486 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0812 09:50:05.697491 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:50:05.697638 (+ 147us) server_negotiation.cc:407] Connection header received
0812 09:50:05.697820 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:50:05.697824 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:50:05.697906 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:50:05.698039 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:50:05.698585 (+ 546us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47508: BlockingRecv error: recv got EOF from 127.0.0.1:47508 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260812 09:50:05.958184 19898 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 09:50:05.957025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47510 (local address 127.13.99.61:39371)
0812 09:50:05.957226 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0812 09:50:05.957232 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 09:50:05.957261 (+ 29us) server_negotiation.cc:407] Connection header received
0812 09:50:05.957422 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:50:05.957427 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:50:05.957502 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:50:05.957579 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0812 09:50:05.958050 (+ 471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47510: BlockingRecv error: recv got EOF from 127.0.0.1:47510 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260812 09:50:06.280603 20071 catalog_manager.cc:1259] Loaded cluster ID: 4cf7ccd296d645ac8fad2cfda1f43d77
I20260812 09:50:06.280692 20071 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5da53722159d430da9d34d90b68294ad: loading cluster ID for follower catalog manager: success
I20260812 09:50:06.282550 20071 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5da53722159d430da9d34d90b68294ad: acquiring CA information for follower catalog manager: success
I20260812 09:50:06.284013 20071 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5da53722159d430da9d34d90b68294ad: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 09:50:06.307699 20119 catalog_manager.cc:1259] Loaded cluster ID: 4cf7ccd296d645ac8fad2cfda1f43d77
I20260812 09:50:06.307783 20119 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 692206eec4d44e4a9360c0d4cf1b97b3: loading cluster ID for follower catalog manager: success
I20260812 09:50:06.309756 20119 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 692206eec4d44e4a9360c0d4cf1b97b3: acquiring CA information for follower catalog manager: success
I20260812 09:50:06.311033 20119 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 692206eec4d44e4a9360c0d4cf1b97b3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 09:50:06.476428 20244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:50:06.472372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47512 (local address 127.13.99.61:39371)
0812 09:50:06.472709 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0812 09:50:06.472713 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:50:06.472729 (+ 16us) server_negotiation.cc:407] Connection header received
0812 09:50:06.472937 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:50:06.472940 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:50:06.473094 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:50:06.473199 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 09:50:06.473458 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:50:06.474116 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:50:06.475012 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:50:06.475228 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:50:06.475303 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:50:06.475584 (+ 281us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 09:50:06.476035 (+ 451us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 09:50:06.476129 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0812 09:50:06.476186 (+ 57us) server_negotiation.cc:299] Negotiation successful
0812 09:50:06.476231 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":101,"threads_started":1}
I20260812 09:50:06.483421 20254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:50:06.479574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41012 (local address 127.13.99.60:37813)
0812 09:50:06.479821 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0812 09:50:06.479825 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:50:06.479838 (+ 13us) server_negotiation.cc:407] Connection header received
0812 09:50:06.479917 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:50:06.479919 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:50:06.480026 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:50:06.480124 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 09:50:06.480318 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:50:06.480971 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:50:06.481848 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:50:06.482100 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:50:06.482213 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:50:06.482537 (+ 324us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 09:50:06.482984 (+ 447us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 09:50:06.483133 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0812 09:50:06.483194 (+ 61us) server_negotiation.cc:299] Negotiation successful
0812 09:50:06.483252 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":97,"threads_started":1}
I20260812 09:50:06.490476 20254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:50:06.486745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41026 (local address 127.13.99.60:37813)
0812 09:50:06.486846 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0812 09:50:06.486853 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 09:50:06.487101 (+ 248us) server_negotiation.cc:407] Connection header received
0812 09:50:06.487214 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:50:06.487217 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:50:06.487285 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:50:06.487359 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 09:50:06.487594 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:50:06.488294 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:50:06.489278 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:50:06.489613 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:50:06.489727 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:50:06.489973 (+ 246us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 09:50:06.490058 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 09:50:06.490127 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0812 09:50:06.490276 (+ 149us) server_negotiation.cc:299] Negotiation successful
0812 09:50:06.490328 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260812 09:50:06.490620 20244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:50:06.486280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47522 (local address 127.13.99.61:39371)
0812 09:50:06.486425 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0812 09:50:06.486430 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 09:50:06.486575 (+ 145us) server_negotiation.cc:407] Connection header received
0812 09:50:06.486792 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:50:06.486795 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:50:06.486850 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:50:06.486899 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 09:50:06.487315 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:50:06.488106 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:50:06.489338 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:50:06.489659 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:50:06.489746 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:50:06.490132 (+ 386us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 09:50:06.490221 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 09:50:06.490291 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0812 09:50:06.490452 (+ 161us) server_negotiation.cc:299] Negotiation successful
0812 09:50:06.490510 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260812 09:50:06.491395 20262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 09:50:06.486245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51212 (local address 127.13.99.62:45545)
0812 09:50:06.486919 (+ 674us) server_negotiation.cc:206] Beginning negotiation
0812 09:50:06.486923 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 09:50:06.486938 (+ 15us) server_negotiation.cc:407] Connection header received
0812 09:50:06.487090 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 09:50:06.487093 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 09:50:06.487260 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 09:50:06.487376 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 09:50:06.487751 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:50:06.488346 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:50:06.489509 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 09:50:06.489846 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 09:50:06.489962 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 09:50:06.490534 (+ 572us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 09:50:06.490885 (+ 351us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 09:50:06.490973 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0812 09:50:06.491132 (+ 159us) server_negotiation.cc:299] Negotiation successful
0812 09:50:06.491201 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":84,"threads_started":1}
I20260812 09:50:06.494315 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 19649
I20260812 09:50:06.510160 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 19771
I20260812 09:50:06.526010 13708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJsh0qK/build/debug/bin/kudu with pid 19901
2026-08-12T09:50:06Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3364 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6715 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (300591 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST